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Abs tract 


This report describes final experiments on pilot decision making 
concluding the work performed under a series of NASA-Ames grants (NAS 2-10047, 
NAG 2-75, NAG 2-112). The focus in this report is on the development of 
models of pilot- decision making in critical in-flight events (CIFE). Analyses 
reported here include development of: 1) a frame system representation 

describing how pilots use their knowledge in a fault diagnosis task (developed 
from an analysis of verbal protocols from an experiment involving twenty 
instrument rated pilots); 2) assessment of script norms, distance measures, 
and Markov models developed from computer aided testing (CAT) data involving 
forty instrument rated pilots; and 3) performance ranking of subject data by a 
group of six recognized expert pilots. 

This research has demonstrated that interactive computer aided testing 
either by touch CRT’s or personal computers is a useful research and training 
device for measuring pilot information management in diagnosing system 
failures in simulated flight situations. Performance is dictated not so much 
by flight hours , ratings and experience with in-flight problems as it is by 
knowledge of aircraft subsystems, initial pilot structuring of the failure 
symptoms and efficient testing of plausible causal hypotheses. 
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I. Background 


A critical in-flight event (CIFE) is a situation that either 
develops quickly or over time. It is unexpected, unplanned, and 
unanticipated , and is perceived by the pilot in command to threaten the 
safety of the aircraft. In the CIFEs studied here, safety of the 
aircraft depends more on the pilot's cognitive processes than on skilled 
motor performance. This report covers the last of three studies directed 
to CIFE research. 

The objectives of the initial project effort were to: 

1) Describe and define the scope of the critical in-flight event 
with emphasis on pilot management of available resources. 

2) Develop detailed scenarios for both full mission simulation and 
paper and pencil (P/P) testing of pilot response to CIFE's. 

3) Develop statistical relationships among pilot characteristics 
and observed responses to CIFE's. 

A subsequent grant focused on the use of computer aided techniques 
to study pilot responses to CIFE's. Using touch CRT's, scenarios 
developed from earlier paper and pencil tests were adapted to computer 
testing using a PLATO®i system. This permitted an efficient, virtually 
experimenter-free study of pilot diagnosis and destination diversion 
decisions . 

The rationale behind the research has remained the same through the 
initial contract and subsequent grant, namely, to understand how pilots: 

_ - 

PLATO is a Control Data Corporation system involving interactive 
computer operations with touch response. 
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a) detect CIFE’s 

b) obtain information on which to base diagnosis 

c) make decisions about destination diversions and 

d) execute these decisions. 


A. Previous Results 1 

Figure 1-1 depicts the overall project accomplishments. 

The initial contract produced: 

1) a test for pilot knowledge of aircraft systems and instrument 
procedures 

2) a series of CIFE scenarios for General Aviation Trainer 
(GAT) simulation 

3) the execution of GAT CIFE experiments 

4) the development of paper/pencil CIFE tests 

5) the development of paper/pencil destination diversion tests 

6) performance measurements of forty instrument rated pilots on 
the above tests. 


1 

For more details on research findings, see ”An Investigation Into 
Pilot and System Response to Critical In-Flight Events," Final 
Report, NASA, NAG2-10047 and NAG2-75, June, 1981. 
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Figure 1-1 


CIFE RESEARCH MILESTONES 


1) Development .of Knowledge Tests of Aircraft Subsystems 

2) Development of CIFE Scenarios for GAT Simulation 

3) Subject Testing with GAT Simulations 

4) Development of Paper and Pencil CIFE Scenarios 

i 

5) Development of Paper/ Pencil Destination Diversion Tests 

6) Testing of Forty Subjects in Paper/Pencil Scenarios 

7) Adaptation of Paper/Pencil Tests to Interactive Computer- 
Aided Testing, Including Knowledge, Diagnostic, and 
Destination Division Tests 

8) Testing of Forty Subjects in Computer-Aided Testing Format 

9) Within Session Learning Studies 

10) Combining Destination Diversion with CIFE Diagnosis in 
Computer-Aided Testing 

11) Descriptive Modeling of Test Results 
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The highlights of the GAT experiments were: 

1) Cockpit managmement style varies widely among pilots, for 
example, some are extremely self-reliant, others want immediate 
and extensive help from ATC, while still others make the 
decision making process a joint effort with ATC. 

2) Good stick and rudder pilots seem to have excess capability and 

maintain good stick and rudder performance before, during and 
after the CIFE. More marginal stick and rudder pilots, on the 
other hand, show increased frequency and amplitude of heading 
and altitude excursions, and experience communication 
difficulties in the face of a CIFE. 1 

3) Pilots who score well on the knowledge test instruments tend to 
perform well in problem diagnosis and decision making. 
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The paper and pencil test results were analyzed with respect to knowledge 
and biographical data. The findings were: 

1) Knowledge is inversely related to total diagnostic inquiries, 
e.g., knowledgeable pilots reach conclusions (right or 
wrong) more rapidly than others. 

2) Total diagnostic inquiries is inversely related to correctness. 
This implies that undirected experimentation is a poor 
diagnosis style. 

3) Diagnosis performance is correlated with knowledge scores. 

4) Knowledge score is correlated with pilot ratings held. 

5) Civil trained pilots place a higher worth on ATC service in 
diversion decisions than do military pilots. 

6) Private pilots place a higher worth on weather factors in 
diversion decisions than do commercial and ATP rated pilots. 

7) ATP rated pilots place high worth on time in 'diversion 
decisions. 

In general the pilots with good diagnostic performance were characterized 
as knowledgeable about aircraft systems, employed few logic tracks to get 
at an answer, used few inquiries per track, and emphasized time in their 
destination diversion decision. They were not differentiated by flight 
hours, ratings, training, or type of flying. 
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B.i Results from the Second Study 1 

In order to avoid experimenter bias and to increase the efficiency 
of CI'FE’ testing, considerable effort was devoted in the second study to 
the adaptation of paper and pencil scenarios to computer aided testing 
using PLATO . Appendix A includes sample displays presented to the 
subject in the course of testing. The router provides the display 
mechanism to select the major program modules, namely: 

1) biographical data 

2) knowledge tests 

3) practice with VORs and Autopilot (to be discussed below) 

4) six CIFE scenarios 

5) the destination diversion scenario 

6) the airport ranking exercise 

7) a combination CIFE diagnosis and destination diversion 
scenario 

The unique advantages of the PLATO system include: 

1) little .experimenter interaction required, hence minimal 

experimenter effects on subject performance 

2) use of touch response to eliminate the need for keyboard 
activation - a special lexicon was developed to assist subjects 
in specifying their diagnoses 


1 

For more details of research findings see "Use of Computer-Aided Testing 
In the Investigation of Pilot Response to Critical In-Flight Events, 
Final Report, NASA, NAG2-112, December 30, 1982. 
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3 ) 


precise records for timing of inquiries and intervals between 
inquiries 

4) the ability to introduce dynamics such as altitude loss during 
a scenario involving power loss 

5) documentation - ability to call up any subject data and to add 
new test data for subjects previously tested 

6) the ability to test subjects at one of the many CDC PLATO 
testing centers around the country with such data made 
available in Columbus for analysis 

Forty subjects have been tested to date using the PLATO approach. A 
list of the scenarios and appropriate diagnoses is shown in Appendix B. 
Typical subject data are shown in Appendix C. Some subjects were used 
for special purposes.* These included: 

1) protocol analysis - having the subjects think aloud during 
their information search 

2) scenario order changes to establish learning within test 
sessions 

3) limiting the number of subject information inquiries while 
relaxing the four minute time constraint 

4) having subjects write out candidate hypotheses as their 
diagnostic search progressed 

While the original thrust of the research involved instrument rated 
pilots, in this PLATO testing phase a small sample of high and low 
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time non-instrument rated pilots were tested as well. This was done to 
estimate pilot rating effect on diagnostic strategies. 

C. Descriptive Modeling for Diagnosis of GIFE’s 

Figures 1-3 and 1-4 depict a way to view subject information seeking 
patterns during diagnosis testing. Sources within logic tracks are 
identified for each scenario. The pilot information plots (PIP’s) are a 
quick way to visualize: 

a) the number of logic tracks employed 

b) the order of inquiries within and between tracks 

c) the time between inquiries 

d) the number of track returns and the information resampled 
Using these PXP's various management information seeking strategies 

can be observed. For the suction failure problem as shown in Figure 1-2, 
Figure 1-3 depicts a subject with a logical and efficient approach to 
diagnosis. Figure 1-4 depicts an almost random inquiry. 

Preliminary findings for the diagnosis portion of the PLATO test 
include : 

1) Subjects are easily motivated using PIATO testing. 

2) Learning within scenario test sessions takes place in terms of 
reduced amount of time between inquiries. The number of repeat 
samples on tracks and the number of tracks sampled are scenario 
dependent . 
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SCENARIO 


You are making a day trip from Augusta, ME to Lebanon, NH. You 
fly out of Augusta at 9:00 a.m. , cleared Victor 39 to Neets 
intersection, Victor 496 to Lebanon. You climb to a cruising 
altitude of 6000 ft. After 15 minutes of routine IMC flying in 
instrument conditions, your instruments indicate an increase in 
airspeed and steadily decreasing altitude while maintaining 
level flight attitude. How would you identify your problem? 

Our Diagnosis of the Problem Was the Following : 

Your vacuum pump failed as indicated by the low reading of the 
suction gauge . The vacuum pump drives the attitude and 
directional gyros. As the artificial horizon lost its drive it 
started to .sag to the right and you compensated by turning 
left, leveling the artificial horizon and putting the plane in 
a slow, descending left bank. The airspeed increase was due to 
the slight nose— down attitude. 

Figure 1-2. Suction Failure Problem and Diagnosis 
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PITOT 



OTHER 


RESPONSE: Gyro Broken 

CONFIDENCE IN RESPONSE: 7 

CRITICALITY: 5 

TOTAL FLIGHT TIME: 100-300 Hrs. 

TOTAL DIAGNOSIS TIME: 78 Sec. 


RATING: Private 

KEY TERM: N 

TOTAL NO. INQUIRIES: 9 

KNOWLEDGE SCORE: 60% 


© 


Figure 1-3. Pilot Information Plot for Scenario #2, Subject #53 
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STATIC 


ENGINE 


ICING 


VACUUM 


OTHER 



RESPONSE: Elevator Control Pressure 

CONFIDENCE IN RESPONSE: 7 

CRITICALITY: 1 

TOTAL FLIGHT TIME: 100 Hrs. 

TOTAL DIAGNOSIS TIME: 211 Sec. 

RATING: Private 


KEY TERM: 

TOTAL NO. INQUIRIES: 20 

KNOWLEDGE SCORE: 502 


Figure 1-4. Pilot Information Pilot for Scenario #2, Subject #67 


3) Instrument rated pilots perform better in general and are more 
efficient than non-instrument rated pilots. 

4) CIFE experiences of test subjects influence their information 
seeking strategies. 

Based on PLATO and paper and pencil test results, idealized 
information searching can be hypothesized. The ideal pilot first 
confirms the symptoms given him. He then establishes whether his engine 
status is threatened by whatever cause lies behind the symptoms. Usually 
oil temperature and pressure and manifold pressure suffice to test this 
condition. Next he generates two or more hypotheses as to the cause, and 
makes a determination of the plausibility of these hypotheses with a 
minimal number of inquires within the appropriate tracks. He rarely 
needs to go over old logic tracks sampled. Finally, given a logical 
cause of the symptoms (usually from the key information element), he will 
often make sure alternative hypotheses are still not viable. 
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D. Combining Destination Division With Diagnosis 

( 

Scenario #5, described in Figure 1-5 was designed to meet two 
purposes : 

1) It was a "no win" problem, i.e., no pilot would really be able 
to find the true cause from the symptoms presented. Hence, it 
would avoid a pilot stumbling into the key information item and 
force the pilot to examine all possible hypotheses. 

2) It combined on the touch CRT both the destination diversion 
decision and the problem of diagnoses. 

Appendix D shows the displays used in advanced PLATO® testing. This 

i 

includes a simulated low altitude chart. A program has been developed 
© 

for PLATO which locates the aircraft relative to the map co-ordinates 
allows heading changes and allows the position of the aircraft to be 
determined from the VOR radials as the aircraft moves at some selected 
speed' and heading. It should be noted that attempts 'to develop pilot 
positional awareness were used throughout. These included position 

t 

reports to ATC based on dual VOR location assessment, auto-pilot heading 
change requirements, new clearances and even a concerned passenger 
requesting a return to the departure airport. Test scenarios employed 
some dynamics, i.e., loss of altitude with neutral forces on the yoke and 
attendant changes in VSI and airspeed with control inputs. As the flight 
progressed, the VOR needle showed a deflection. The scenario permitted 
communication with ATC and allowed for declaring emergencies'. 



Scenario #5 


Consult attached simplified low altitude chart. 

You are' on' an IFR flight from Utah Municipal Airport to Haven County 
Airport. You depart on V-110 at 6000 ft. in your Cherokee Arrow (N123B) which 
is equipped with a 3-axis autopilot. There is a NOTAM out which reports that 
Colorado VOR is out of service during the period you plan to navigate. 

Navigate using Ohigh and California VORs . You have been enroute 60 minutes 
from Utah Municipal Airport. You are on the gauges but the ride is smooth. 
Weather briefing indicated that winds at 6000 were expected to be light and 
variable . 

You have one passenger aboard. 

Weather at: 

Haven County Airport = 2000 & 5 
Ohigh = 1000 & 3 

Wind Falls = 1000 & 3 by a C-172 
( 10 Minutes Ago ) 

Cleve Center calls and reports radar contact is lost. Please report present 
condition. 


Clearance 


ATC Response: 


N123B, thanks for the position report. 

Here is your new clearance: 

proceed direct California VOR direct 
Haven Count Airport at 6000. 

There will be opposite traffic at 5000 .... maintain 6000. 
Please confirm your new heading and altitude after your turn. 

Scenario Change 


While practicing hand flying with your autopilot disengaged, you notice that 
increased nose-up trim is required to maintain a constant indicated altitude 
and that your IAS has decreased 20 kts. from normal cruise. 

Your passenger notes this problem, and suggests that you turn back to Utah 
Municipal. 

Determine the nature of the problem, and your destination decision. 


Figure 1-5. Combined Diagnosis and Destination Diversion Scenario. 
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Early tests suggest most pilots were preoccupied with diagnosis and 
seemed to lose positional awareness.’ This result is supported by typical 
experiences of pilots and ATC personnel. 

E. Destination Diversion Descriptive Modeling 

Appendix E presents the CRT displays given to the subject for the 
destination diversion phase of testing. In this scenario the pilot loses 
his alternator in IMC conditions beyond the range of his destination or 
departure airports. Under time constraints he must choose between 
alternate airports with different attributes. Such attributes include 
ceiling, visibility, bearing and distance, navigation aids, presence of 
ATC support and terrain. In this test there is no absolute answer. 

Rather the test seeks to see how pilots weight attribute information 
about alternative airports. 

Table 1-1 summarizes for twenty-two subjects the attributes selected 
by airport. Note ceiling, visibility, and approach aids dominate the 
inquires . 

These graphs, called DIG's for destination information graphs, give 
a quick picture of how pilots evaluate diversion airport attributes. 

There appears to be no biographical or knowledge predictors for these 
graphs. The relationship of DIG's to CIFE PIP's will require further 
s tudy . 
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Table 1-1. Frequency of Information Type for Each Airport. 
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II. Research Objectives for the Current Study 


The basic aim of this third study was to add depth of understanding 
to pilot information seeking and decision making in the face of critical 
in-flight events. The past research program sought to apply human 
factors concepts to pilot information processing and decision making in 
order to: 

a) ascertain the role of pilot background, experience and 
knowledge in problem diagnosis and decision making, 

b) describe problem solving paths in sufficient detail to permit 
the identification of various general strategies used by 
pilots, and 

c) identify generalizable formal structures to describe CIFE 
decision making and diagnostic behavior. 

Experiments in the first two studies resulted in a wealth of data 
from some eighty pilots who were subjected to a set of diagnostic and 
decision making scenarios. Those data have been used to develop simple 
graphical models to depict information gathering behavior and have been 
subjected to an exhaustive set of statistical tests to demonstrate 
relationships between pilot backgound and scenario test scores. The 
major task remaining for this, the third phase, was to develop formal 
structures to generalize behaviors observed in the graphical models 
(PIP's) and more fully exploit available data. 
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To accomplish a general description of pilot diagnostic behavior 
concepts from cognitive psychology and artificial intelligence will be 
employed as well as more traditional mathematical models from the theory 
of stochastic processes. 

Over the last decade, significant strides have been made toward 
understanding how people acquire and utilize knowledge. Contributions 
from the fields of cognitive psychology and artificial intelligence 
emphasize the critical role that the organization of knowledge in the 
mind plays in determining performance. Of particular interest to 
understanding the diagnostic behavior of pilots are the (closely related) 
concepts of frames (Minsky, 1975), schemata (Thorndyke and Hayes-Roth, 
1979) and scripts (Schank and Abelson, 1977). 

The goal of this research is to develop a more formal description of 
a pilot's diagnostic behavior. An operating hypothesis is that a pilot’s 
behavior in response to a given scenario will be mediated~in part by the 
relevant knowledge structures contained in that pilot's memory. This 
hypothesis suggests three basic research questions: 

1) What knowledge does a pilot have? This might be thought of as 
an unordered list of propositions. 

2) How is this knowledge organized in memory? Models of human 
associative memory (Anderson and Bower, 1980) and 
schemata-based theories seem most promising in answering this 
question. 
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3) How are knowledge and organizing knowledge structures used by 
pilots in attempting to diagnose the causes of critical 
in-flight events? Commonly hypothesized cognitive structures 
and processes such as control elements , activation thresholds , 
push-down stacks, and priming may prove to be applicable. 
Concepts borrowed from artificial intelligence research, such 
as production and frame systems, may be useful. 
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III. The Role of Knowledge Structures in Fault Diagnosis 


The study di-scussed below addresses the question of how 

domain-specific knowledge is used in fault diagnosis (Rasmussen and 

Rouse, 1981). The fault studied is the failure of the vacuum system in 

an airplane. The domain-specific knowledge of interest is the knowledge 

that instrument rated pilots have stored in their memories. 

It is hypothesized that such knowledge is organized in a pilot's 

memory as a frame system. The basic knowledge structure within such a 

.system is a frame. Minsky (1975) defines a frame as: 

"a data structure for representing a stereotyped situation, like 
being in a living room or going to a child's birthday party. 

Attached to each frame are several kinds of information. Some of 
this information is about how to use the frame. Some is about what 
one can expect to happen next. Some is about what to do if these 
expectations are not confirmed” (p. 212) 

In an aviation setting one frame or "stereotyped situation" might be 
a plane in a descent. Such a data-structure would contain information 
about expected instrument readings (e.g., the altimeter should show 
decreasing altitude). It could also contain labeled slots indicating 
relevant pieces of information that should be collected (e.g., what is 
the indicated airspeed?).' Associated with each such slot (Winston, 1984; 
Schank and Riesbeck, 1981) is a set of permissible slot-fillers (e.g., 
increasing, constant or decreasing airspeed). 

If a slot has several alternative slot-fillers, one of these may be 
marked as the default value. It is the value most likely to be correct 
if the situation described by the frame occurs. 
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Within this framework, fault diagnosis consists of: 

1) Focusing attention on a particular frame. 

2) Using the knowledge contained in that frame to generate 
information requests and to make inferences. 

3) Using the information collected to confirm or reject the 
activated frame as an appropriate representation of the 
situation. 

4) Using the contents of the frame to activate or focus 
attention on additional relevant frames . 
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METHOD 


Pilots were read a scenario that provided certain instrument 
indications and background information pertaining to a flight over the 
New England area. They were told that a problem existed at the point 
in time described by the scenario, and were asked to try to diagnose the 
cause of the problem. In order to perform this task they were allowed to 
request any information that would normally be available to a pilot under 
the conditions specified by the scenario. Requested information was 
provided verbally by an experimenter. 

Each subject was tested in a separate session. The entire session 
was tape-recorded . 

Subjects 

Twenty— six pilots with instrument ratings served as subjects. 

Pilots were paid $10 for a single session that lasted from one to two 
hours. Additional biographical information is summarized in the Results 
and Discussion section. 

Procedure 

The two primary tasks involved fault diagnosis (Task 1) and a memory 
(recall) test (Task 2). In addition, four other supplementary tasks were 
performed. Tasks were run in numerical order for all subjects. Since 
Tasks 1 and 2 were always run first, there is a potential for confounding 
of results on the remaining tasks. Consequently, the information 
provided by these latter tasks will be presented only in so far as it 
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supports or contradicts conclusions drawn from the two primary tasks. 
Similar caution must be applied when interpreting results from Task 2. 

Task 1 . Each subject was asked to describe what "a pilot should do 
in order to determine the cause of a problem that has developed while 
flying a Cherokee Arrow, with a 200 horsepower, fuel-injected 
Lycoming engine. This particular plane is not turbocharged and does 
not have an autopilot." 

The subject was asked to think out loud as he tried to diagnose the 
problem, telling the experimenter not only what information a pilot 
should collect but also why he should collect that information. 

A sample verbal protocol was read to the subject in order to 
illustrate what was meant by thinking out loud. This example involved 
deciding how to get to work rather than fault diagnoses while in flight. 

The pilot was then asked to practice thinking out loud using a 
scenario involving selection of a restaurant. Thus, this practice did 
not deal with fault diagnosis or with flying. 

Pilots were told specifically not to try to correct the problem. 
Their sole objective was to dete rm ine the cause of the problem. It was 
pointed out that this was therefore a different task from the one faced 
by a pilot who was actually flying under such conditions . 

Pilots were further informed that this was a static "simulation," 
that the information provided in response to their requests referred to 
the same point in time at which the ‘scenario was read. (The plane did 
not continue to fly during the time taken to diagnose the problem.) 

Quantitative readings were provided in response to information requests 
whenever appropriate (e.g., the vertical speed indicator shows a descent 
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at a rate of 600 feet per minute). Whenever appropriate, it was also 
stated whether the quantitative reading would be a normal one for the 
plane while cruising at the intended altitude (e.g., the oil temperature 
gauge reads 140°F, which is normal). 

The responses provided were developed by an expert pilot and listed 
for the experimenter to read in response to a subject’s requests. The 
accuracy of the responses was tested by simulating the scenario conditons 
in an actual flight. 

Two experimenters were present for each session. One provided the 
responses to the information requests. The other gave the instructions 
and prompted the subject to continue thinking out loud when necessary. 

The prompts consisted of two questions: 

1. What are you thinking now? (if the pilot became silent) 

2. Why are you interested in that information? (if the pilot 
asked for information without explaining why he wanted it) 

The scenario (presented below) was one in which a plane's vacuum pump 
failed. This fact was indicated by a zero reading on the suction 
gauge. The vacuum pump drives the artificial horizon and directional 
gyro. As the artificial horizon lost its drive, it started to sag to 
the right and the pilot compensated (unconsciously) by turning left. 

This leveled the artificial horizon and put the plane in a descending 
left bank. The resulting nqse-down attitude caused an increase in 
airspeed and a descent. 

At the point in time represented in the scenario the plane 
had faulty readings on the artificial horizon and directional gyro. The 
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plane was descending nose-down and was in a left bank while these 
instruments indicated straight and level flight. 

The scenario is given below. 


Imagine that this pilot is making a day trip from Augusta, Maine to 
Lebanon, New Hampshire. He flies out of Augusta at 9 a.m. cleared 
Victor 39 to Neets intersection, Victor 496 to Lebanon. He climbs 
to a cruising altitude of 6000 feet. 


After 15 minutes of routine flying in instrument conditions in the 
clouds, the instruments indicate an increase in airspeed, a steadily 
decreasing altitude, and zero pitch. 


So, the instruments indicate an increase in airspeed, a steadily 
decreasing altitude, and zero pitch. 


How should this pilot go about identifying his problem? 

After hearing the scenario, pilots began requesting information in 
an effort to diagnose the fault. They continued until they arrived at a 
conclusion or decided that, with the information available, it was 
impossible to arrive at a conclusion. 


Pilots stating a conclusion were asked to rate their confidence in 
it on a scale from one to ten. A rating of one indicated the lowest 
level of confidence while ten indicated the highest. 

Task 2 . Pilots were asked to recall everything they remembered about 
this flight. They were told to be very specific about any instrument 
readings or conditions they remembered . 

Task 3 . Pilots were given a knowledge test. They were asked to 
describe the information they would collect in order to determine whether 
a plane had one of the following problems : 
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1 . Structural icing 

2. Directional gyro malfunction (mechanical failure) 

3. Suction gauge malfunction (mechanical failure) 

4. Vacuum pump failure 

5. Iced-up pitot tube 

6. Accidentally extended gear 

7. Iced-up static port 

8. Accidental change of the yoke position (with the pilot unaware 
of this fact) 

9. Artifical horizon malfunction (mechanical failure) 

10. Reduction in power or thrust from the engine or prop. 

They were told to assume the plane was cruising in the clouds at the time 

one of these problems occurred. 

Task 4 . Pilots were asked to think back to the original 
problem-solving task. They were asked to describe their impression of 
the plane’s physical orientation while they were trying to diagnose the 
problem. 

Task 5 . Pilots were asked whether they formed a visual (mental) 
image of an instrument panel while performing Task 1. If the answer was 
yes, they were asked what they visualized. 

Task 6 . Pilots were asked to provide the following biographical 
information: 

1 . Whether they had ever personally experienced problems 
with icing or the vacuum system while flying 

2. Age 
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3. Total hours of flying experience 

4. Highest airman's certificate 

5. Ratings in addition to instrument rating. 


RESULTS AND DISCUSSION 


Before presenting summary statistics for the results of the 
experiment, three full verbal protocols will be presented: 

Subject #17 

Query 1: "What if I open my alternate static source?" 

[Why? ] "To see if I have a clogged port.". 

Query 2: "You said the airspeed was increasing and the altitude 

was decreasing?" 

[Why? ] "That suggests that you're descending." 

Query 3: "What is the outside air temperature?" 

[Why? 3 "Because there could be ice on the wings 
causing descent." 

Query 4: "What is the RPM?" 

[Why? 3 "To see if there’s a loss of engine power." 

Query 5 : " Can you see ice on the wings ? " 

Conclusion: "I can't tell." 

Subject #3 

Query 1: "Steadily decreasing altitude. Then I would also assume 

that that also includes then a showing a descent on the 
vertical velocity indicator?" [Why?] "Is the vertical 
speed indicator having a reading consistent with the 
altimeter. To try to narrow down is it a pitot-static 
system problem." 

"That indicates to me that the vertical speed indicator is 
consistent with the altimeter." 

Query 2: "To clarify. Is my, the altitude ’ indicator also going 

down?" 

Query 3: "At this point, then, I would then change my attention 

away from the, no I take that back. The airspeed 
indicator is indicating an increase in airspeed. Is that 
correct?" 

"At this point, I will rule out the pitot-static system. 
Those instruments all seem to be consistent." 
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Query 4 
Query 5 

Query 6 

Query 7 
Query 8 


Conclusion: 

Subject #1 

Query 1 


"With an increase in airspeed, then, the next question is, 
is the manifold pressure, what is the trend of the 
manifold pressure gauge?" [Why?] "To try to narrow down 
is it an engine problem of some sort, am I losing engine 
power . " 

"Also with regard the engine just to get information as 
to whether the engine and prop in this case is working 
correctly, what is the EPM reading?" 

"At this point it seems that the pitot-static system is 
correct. The engine seems to be functioning correctly. 

The engine seems to be running, producing power." 

"My next line of thought would be some sort of control 
problem. I was going to ask a question about the trim, 
but I'm assuming the trim hasn't been played with. I 
just, a new thought came to mind and the new thought is 
that if I am decreasing altitude and zero pitch change, in 
other words, I haven't evidently put in any control 
input to affect the elevator. Well, let me phrase it as a 
question. Is there ice, am I receiving ice on the wings 
of any sort?" 

"That takes care of that problem." 

"Then let's go back to the controls. Is the pitch trim 
operating correctly, the trim wheel? Has the trim wheel 
changed position?" 

"At this point I'm becoming stumped. Let me ask another 
question which maybe clarifies the initial conditions. 

That is, I have zero pitch, meaning that indicates that I 
haven't had a forward deflection in the control wheel. I 
haven't added down elevator. I’m losing altitude, gaining 
airspeed, but have not had a, is the nose pitched over is 
what I'm trying to determine at this point. I'm in the 
clouds . The only way to determine that is either through 
the altitude change, which obviously is down, but the next 
thing to check would be the attitude indicator and I'm 
assuming that the attitude indicator is indicating level 
because the initial condition saying there was no pitch 
change. Ah! I have just rung a bell! Next question: Is 

the vacuum, what is the reading on the vacuum gauge?" 

"My problem is with the vacuum system and I'm losing 
pressure to my gyroscopic instruments."- 


: "The first thing I would think about is with the 

decreasing altitude and increasing airspeed, that for some 
reason the plane is starting to go down and I would look 
to confirm that right away with the attitude indicator. 
There is zero pitch in there. It should show down pitch. 
So the first instrument I would look at since it runs off 
of suction, would be over at the, suction,, to see if it's 
producing any vacuum. What- does the suction gauge show?" 
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Conclusion: "You have a nose-down attitude and the vacuum pump's 

gone . ” 

These three protocols illustrate the apparent heterogeneity of the 
subjects' performances. Subject #1 asked one question while Subject #3 
asked eight, yet both arrived at the same conclusion. Subject #17 made 
five queries, none of which matched the query made by Subject #1, and 
decided he could not determine the cause of the problem with the 
available information. 

Although data were collected for twenty-six subjects, the following 
analysis will be based on only twenty of these pilots. Since the 
objective of this study was to model the way pilots use their knowledge 
structures (as opposed to whether they have the necessary knowledge), 
any subject demonstrating knowledge errors (Task 3) that would prevent 
him from solving the problem was deleted from the data set. Four 
subjects were deleted for this reason. All four thought the vacuum 
system was connected to the static port. One of these _ four thought the 
altimeter and airspeed indicator were vacuum driven instruments. Another 
did not know which instruments were part of .the vacuum system and which 
were part of the pitot-static system. 

Not surprisingly, three of these four subjects concluded that it was 
impossible to determine the cause of the problem. The fourth decided 
that the problem was a malfunction of the artificial horizon. 

All four of these pilots had between 500 and 2000 hours of flying 
experience and were between the ages of 41 and 70. Three had private 
pilot licenses and one a commercial license. (All subjects in the 
experiment were current instrument rated pilots.) 
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A fifth subject was deleted for failing to follow instructions. He 
insisted on trying to fly the plane in order to halt the descent, rather 
than attempting to determine what was causing the problem. 

A sixth subject was eliminated because he misinterpreted the meaning 
of the scenario. He interpreted zero pitch to mean that the vertical 
speed indicator showed zero and the artificial horizon indicated the 
plane's nose on the horizon. This misinterpretation was corrected when 
it was discovered (after his first query). He was then allowed to 
proceed , and concluded that the cause of the problem could not be 
determined. 


His comments after hearing the solution to the problem at the end of the 
experiment are worth noting: 

"I had it [the scenario events] happen. It’s interesting that I had 
it happen and I didn't relate to it. But, so you were getting a 
correct indication from the airspeed and the altimeter. You were 
incorrect from the attitude indicator and you were actually in a 
turn. If I'd checked my heading, I'd have been alright. ... When I 
lost the attitude indicator [in the actual flight], it didn't feel 
right in the airplane. I felt almost like I was getting vertigo 
because I was in IFR conditions and kind of in and out of the clouds 
and things just didn't feel right. I was still following that dumb 
attitude indicator. Then I noticed I was off my heading, there was 
something wrong, and I looked and the suction gauge was showing 
zero." 

So, the scenario can happen. 

Summary Statistics 


Of the twenty final subjects, eleven concluded there was a vacuum 
system failure (Group A), four stated that the problem was a 
malfunctioning artificial horizon (Group B) , one decided the problem was 
a downdraft (Group C) , and three concluded that the problem could not be 
diagnosed with the available information (Group D) . A final pilot (Group 
E) detected the faulty artificial horizon but then concluded that he 
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could not diagnose the problem (for reasons that will be explained 
later). Certain characteristics of these pilots are summarized in Table 
3-1 (by group) . 

The conclusions of the four groups are summarized in Table 3-2, 
along with the pilots' confidence ratings. 

In Task 4, pilots were asked what their final conclusion was 
regarding the plane’s physical orientation (at the end of the Scenario 
Task) . Fourteen reported that they thought the plane was in a straight 
nose-down descent. Five (all in Group A) thought the plane was 
descending in a left-bank with the nose down. The pilot in Group C 
thought the plane was in a straight and level descent, with the nose on 
the horizon. 

In the recall task, the question of interest is whether the 
pilots remembered the three instrument indications given in the scenario. 
Assuming that the probability of retreiving information from memory is 
related to the amount of attention it was given during the problem 
solving task, this provides evidence of the salience of these three 
instrument readings. The results are summarized in Table 3-3. Note 
that none of the subjects in Group D recalled the zero pitch 
indication. Below is the full response of one pilot in Group D to the 
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recall task. He failed to recall the pitch indication despite his 
overall high recall performance. 


"You were first cleared from Augusta, Maine to Neets intersection 
via Victor 23 to Lebanon, New Hampshire at 6000 feet in instrument 
conditions at 9:00 a.m. You explained that we were showing an 
increase in airspeed, an airspeed of 140 and increasing, an 
altitude of 5600 and decreasing and a vertical speed decreasing at 
600 feet per minute. I then asked what the manifold pressure was 
and you said 22.5 inches, and I asked RPMs. You said it was 2300 
RPMs. I asked about the control column and what position it was and 
you said it was in the neutral position. I also asked about the 
trim tab and he explained to me that the trim tab was set for level 
flight. I asked him if he could see any icing on the wings and he 
said we couldn't see any. I also asked, I think that's basically 
all I asked." 


A more detailed presentation and discussion of results is given below. 
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Table 3-lb. Biographical Data (continued) 

Ratings! In Addition to Instrument 
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Table 3-lc 


Biographical Data (continued) 
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Table 3-2. Conclusions at the End of the Scenario Task 


Conclusion 


Group 

X/ 

Vacuum System Malfunction 

Artificial Horizon 
Malfunction 

Downdraft 

Can't Tell 

Confidence 

Rating* 





- 

6_ 

7 

8 

9 

10 

A 

11 

0 

0 

0 

1 

0 

0 

4 

6 

B 

0 

4 

0 

0 

0 

1 

1 

1 

1 

c 

0 

0 

1 

0 

0 

0 

0 

1 

0 

B 

0 

0 

0 

3 

- 

- 

- 

- 

- 

E 

0 

0 

0 

1 

- 

- 

- 

- 

- 


*10 indicates the highest level of confidence in a conclusion. 1 indicates the lowest level 
of confidence, h'o rating was stated if the pilot concluded he could not tell what the 
problem was. The dashes indicate that no confidence rating was given because the pilot 
failed to arrive at a diagnosis. 
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Table 3-3. Recall of the Three Original Instrument Indications (Task 2) 


Group 


Increasing Airspeed Decreasing Altitude Zero L’itch Number of Subjects 


A 
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The other subject did recall a descent indicated- on the vertical speed indicator. 
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Preview 


In the analysis to follow, verbal protocols have been used to help 
identify patterns in the pilots' information requests and diagnoses. 
These patterns have then been used to make inferences about the 
underlying mental processes and structures (Gentner and Stevens, 1983). 

Some strong assumptions have been made in order to map out a 
parsimonious general model of performance. When made, they will be 
explicitly stated. The impact of these assumptions upon the final model 
will be evaluated at the close of the discussion. 

Initially Activiated Frames • 

Before asking how frames are activated in the memory of a pilot , we 
must first determine what frames are being activated. Three sources of 
data are relevant to this question: 

1 . The spontaneous comments made by the pilot 
before making his first query. 

2. The first query made by the pilot. 

3. The statements made by the pilot immediately 

after his first query (either spontaneously or in 
response to the experimenter's prod: Why are 

you interested in that information?). 
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The following assumptions will be made to infer the existence of 
particular frames : 

1* Frames are prototypes (Aikens, 1983; Sowa, 1984) representing 
states of nature (e.g., the plane is descending or 
the static port is blocked). 

2. Queries (e.g.. What does the altimeter show?) are generated 
by activated frames. Thus, the questions a pilot asks 
should provide insights into states of nature 
represented by the activated frames. 

3. The activated frame provides the answer to the question: 

Why are you interested in that information? (See Schank 
and Riesbeck, 1981; Winston, 1984.) 

Suppose there exists a frame labeled DESCENT that has a slot 
indicating an expected reading on the altimeter. When activated, this 
frame directs the pilot to determine whether all expectations are met 
(see Figure 3-1). Thus, the instructions for use (Minsky, 1975) direct 
the pilot to ask: What does the ALTIMETER show? If the pilot is then 

asked why he is interested in this information, he looks at the label for 
the activated frame and responds: Because I want to see whether the 

plane is in a DESCENT. In this manner ,' frames are used to generate and 
to respond to questions. 
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Frame Label: DESCENT 


Expectations: DECREASE SHOWN ON ALTIMETER 

Instructions for Use: CHECK TO SEE WHETHER 

EXPECTATIONS ARE MET 

Figure 3-1. Sample Frame. 
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Consider the following verbal protocol. 


Pilot : 

"Steadily decreasing altitude. Then I would also assume that 
that also includes then a showing a descent on the vertical 
' velocity indicator. What does the vertical speed indicator 
show?" 

Experimenter ; 

"Why are you interested in that?” 


Pilot : 

"Is the vertical speed indicator having a reading consistent 
with the altimeter to try to narrow down is it a pitot-static 
problem?" 


From this data, we would infer the existence of a frame labeled 
P.ITOT-STATIC SYSTEM MALFUNCTION. 

Table 3-4 shows the initial queries for the 20 pilots studied. 
Based on an analysis of the associated verbal protocols, labels for ten 
frames were identified: 

1. The Plane is in a DESCENT. 

' Three subjects appeared to generate their first queries using 
this frame. Two asked what the vertical speed indicator was 
showing. The other asked about the tachometer reading. 
Comments supporting this inference included: 

"If it’s [the vertical speed indicator’s] reading correctly 
it’ll read whether you’re descending," 
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"Is it [the plane] going down? 


"We seem to be in a descent." 

2. There is a POWER LOSS. 

Four pilots activated this frame, three inquiring about the 
manifold pressure gauge and one about the tachometer. 

Associated comments indicating that the POWER LOSS frame had 
been activated included : 

"What's happening to my power? Very definitely we have a 
situation where we seem to be losing power." 

"Let's find out what the engine is doing." 

"I'm going down and what I have to do is figure out why I'm 
going down. I'd look for power.” 

3. There is ICING. 

One subject used this frame to generate a question about the 
presence of visible moisture "because it might be in regards to 
icing". 

4. There is a PITOT-STATIC SYSTEM MALFUNCTION. 

Two subjects requested the reading on the vertical speed 
indicator after activating this frame. A third asked about the 
outside air temperature. 

Consistent statements included the sample subject discussed 
- earlier who wondered: 
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"Is it a pitot-static problem 


Other comments were: 


"It appears it could be a problem with the pitot-static 
system." 


"My first thoughts are that there's definitely going to be a 
problem with the pitot-static system.” 


5. There is STATIC PORT ICING. 

This subject inquired about the reading on the outside air 
temperature gauge, stating: 


"The first thing I would think of would be a problem with the 
static port because I'm thinking it could be a possibility of 
something freezing." 


6. There is a BLOCKED STATIC PORT. 

Both of these pilots wondered what would happen to the 
instrument readings if the alternate .static source was opened 
because they were concerned with a blocked static port: 

"There may be a blockage in the static system that's causing 
faulty instrument readings." 

[There may be a] "clogged port." 


7. There is PITOT TUBE ICING. 

This subject asked if the pitot heat was on, saying: 
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”1 was wondering if there might be in that area [the pitot 
tube] icing. Picked up some icing." 

8. There is an AIRSPEED INDICATOR MALFUNCTION. 

This pilot checked to see whether there was in increase in air 
stream noise as: 

"This would indicate an increase in airspeed, would back up 
that instrument indication." 

9. There is a VACUUM SYSTEM MALFUNCTION. 

Two pilots asked for the suction gauge reading to: 

"see if it is producing any vacuum." 

10. My MEMORY may be in ERROR. 

Two subjects seem to have a frame dealing with beliefs about 
their own limitations and abilities (Norman, 1983). This frame 
is concerned with the possibility that the pilot has not 
recalled the scenario information correctly. The first query 
is intended: 

"to make sure I’m right on the scenario." 

One subject asked whether the airspeed indicator showed an 
increase (it did). The second subject asked whether his 
airspeed indicator was showing a decrease. 
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Table 3-4. Initial Queries 


Number of Subjects 
Asking 


What is the reading on the: 

Vertical Speed Indicator? 4 

Airspeed Indicator? 2 

Manifold Pressure Gauge? 3 

Tachometer? 2 

Outside Air Temperature Gauge? 2 

Suction Gauge? 2 

What happens if : 

The Alternate Static Source is Opened? 2 

Is : 

The Pitot Heat On? 1 

There Visible Moisture in the Air? 1 

There an Increase in Wind Noise Outside 

the Plane? 1 
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The above cited evidence indicates that a variety of frames exist in 
pilots’ memories. Furthermore, it suggests that the same "stimulus" 
(reading of the scenario) can lead to the activation of different frames 
in different pilots’ memories . In some cases the frames that have been 
activated are very similar to one another (e.g., STATIC PORT ICING and 

BLOCKED STATIC PORT). In other cases, however, they are radically 
different (e.g., VACUUM SYSTEM MALFUNCTION versus TOWER LOSS). 

Initial Activation of Frames 

The above analysis identified ten frames that were used by the 
twenty pilots to generate their initial queries. (Not all pilots 
activated the same frame..) This subsection addresses the next question: 
How were these frames activated? The goal is to better understand the 
mental processes that occurred between the time the experimenter began 
reading the scenario and the time at which the pilot made his first 
query. 

By its very nature, protocol analysis provides only fleeting 
glimpses into the mental processes occurring within any one subject. 
Subjects do not report all of their thoughts. Furthermore, even if two 
subjects activate the same set of mental processes, their comments may 
provide evidence relevant to different portions of these processes. 

Thus, in order to construct a model that is even somewhat complete, it is 
desirable to make an assumption: 
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Unless evidence to the contrary exists, assume that if two pilots ask the 
same question (e.g., What is the reading on the manifold pressure 
gauge?), the same mental processes (at least in terms of important 
characteristics) produced that query. This assumption is based on an 
objective of developing a parsimonious explanation of performance (a 
desire to introduce individual differences only when necessary) . Its 
implications and applications will be made clearer in the following 
analyses . 

The scenario that was read can be thought of as a set of cues or 
clues indicating what the problem was. The first questions to be 
addressed are: 

1. What are the cues that subjects are attending to? 

2. What frames are being activated by these cues? 

(Pauker, Gorry, Kassirer and Schwartz, 1976.) 

Evidence that a cue has been given attention is the fact that 
the pilot repeats it out loud. There may, of course, be other cues that' 
have received attention, but that the pilot has not repeated. Consider 
the following comments made by pilots after hearing the scenario: 

S#l. ."The first I would think about is with decreasing altitude and 
increasing airspeed, that for some reason the plane is 
starting to go down.*' 

S#2. "It would appear that he's descending." 

S#5. "If increasing airspeed, altimeter unwinding, those are pretty 
good clues he's descending." 

S#6. "I'm going down and what I have to do is figure out’ why I'm 
going down. I'd look for power." 
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These protocols contain evidence regarding the activation of two frames 
(DESCENT and POWER LOSS) resulting from attention to two cues (increasing 

airspeed and decreasing altitude). Based on this data alone, the most 

parsimonious model is one in which: 

1. Two cues, an increase shown on the airspeed indicator and 

a decrease shown on the altimeter, activate the DESCENT frame. 

2. The Instructions for Use in the DESCENT frame direct the pilot 

to look for possible causes of descent, resulting in activation 

of the POWER LOSS frame (see Figure 3-2). 

The comments by Subjects #1 and #5 both provide evidence regarding 
their attention to these two external cues and the activation of the 
DESCENT frame. For the purpose of developing a general model, it is 
assumed that Subject #2 also attended to both cues and activated the 
DESCENT FRAME, even though complete evidence to support this is lacking. 
(There is no contradictory evidence.) Subject #6 provides evidence that 
two frames were activated, the DESCENT frame followed by the POWER LOSS 
frame . It is assumed that he attended to the two external cues shown in 
Figure 3-2, even though he provides no evidence of this (i.e., it is 
assumed that his behavior was like that of the other subjects since no 
evidence to the contrary exists). 

This type of analysis will now be applied to the data for all twenty 
subjects. The data to be used for the following analysis will be the 
spontaneous comments of a pilot before his first query, his first query, 
and his comments immediately after the first query (spontaneous or in 
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response to the prod: Why are you interested in that information?). 

Thus, the data to be used consists of all statements made after the 
reading of the scenario, but before the asking of a second query by the 
pilot. 

Nose-Down DESCENT . Figure 3-3(a-d) shows the model constituents 
supported by the data for each of seven subjects. Associated comments 
are also quoted to indicate the support for model constituents. When the 
data for a given subject provides no evidence in support of a given model 
constituent, that constituent has not been filled in. 

Figure 3-3 indicates that these seven subjects activated 
three types of frames (DESCENT, POWER LOSS and VACUUM SYSTEM 
MALFUNCTION) . A total of six variants on these three types of frames 
was observed. 

The general model of performance assumes that all seven subjects 
activated the DESCENT frame first because they focused their attention on 
two cues, a decrease on the altimeter and an increase on the airspeed 
indicator. When asked later (Task 4) what they believed the plane's 
orientation to be at the beginning of the scenario task, all seven 
subjects stated that it was in a straight, nose-down descent. Comments 
included : 

"Descending like this [straight descent shown with hand] in a 
nose-down attitude." 

"Descending straight ahead, slight nose-do™." 

"Straight, nose-down." 
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Figure 3-2. A Partial Model of the Frame Activation Process 
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Frame Label: DESCENT 


External Cues* 


Expectations: DESCENT INDICATED ON VERTICAL SPEED INDICATOR 

Instructions: CHECK TO SEE WHETHER EXPECTATIONS ARE MET 


Ul 

H 


Supporting Comments: S/12 - "It would appear that [the plane's] descending. 

What does the vertical 6peed indicator show?" 


S#10 — "What does the vertical speed Indicators show?” [Why?) 

"If it's reading correctly it will read whether you are 
descending." 


Figure 3-3a. Subjects #2 and #10. 


(Note: No evidence was present regarding external cues. 

Therefore, this part of the system was not filled in for 
Subjects S2 and #10). 
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Supporting Comments: S#4 - "I'm going down and what I have to do is figure out why 

I'm going down. I'd look for power. What is the manifold 
pressure?" 
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SW - "What's the manifold pressure read?" {Why? J 
“Find out what the engine is doing." 


Figure 3~3b, Subjects #4 and #6 
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Supporting Comments: "What is the RPM?" (Why?] 

"We seem to be in a descent." 


Figure 3-3c. Subject #18. 


*It is probable that this subject has activated the POWER LOSS 
frame as a possible cause of DESCENT. There is no direct evidence 
of this in the verbal protocol, however. Consequently, these 
portions of the frames have been left blank. 
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Supporting Comments: "The first I^would think about is with the decreasing altitude and 

tj, increasing airspeed, that for some reason the plane is starting 

•£~ to go down and I would look to confirm that right away with the 

attitude indicator [the arcifical horizon]. There is zero pitch 
in there. It should show down pitch. So the first Instrument X 
would look at, since it runs off of suction, would be over at 
the suction gauge, to see if its producing any vacuum." 
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"If increasing airspeed, altimeter unwinding, those are pretty good 
cues he's descending and that he's got a nose-down attitude. But 
since it's not being indicated on the gyro horizon there's a pretty 
good chance that he's lost his vacuum. What does the vacuum gauge show?" 

Figure 3-3d. Subjects #1 and #5. 


Figure 3(a-d). Activation of Initial Frames for Seven Subjects Focusing on a Nose-Down Descent 





This assumption makes sense in terms of the DESCENT frame as defined. If 
the plane is in a straight nose-down descent, the airspeed will 
increase and the altitude will decrease. 

Having activated the (nose-down) DESCENT frame., four subjects 
elected to "test the hypothesis" that this was the correct frame (was a 
valid model of the state of nature) by checking to see whether the 
instrument indications were consistent with this model. A descending 
plane should show a DESCENT INDICATED ON VERTICAL SPEED INDICATOR, so two 
subjects checked this expectation (see Figure 3-3a) . The other two 
subjects checking for instrument indications looked for an 
indication of PITCH DOW ON ARTIFICIAL HORIZON (see Figure 3-3d). Since 
this information was available in their (short-term) memories, they 
answered the question without requesting this information from the 
experimenter. This expectation was found to be invalid (the artificial 
horizon showed zero pitch), suggesting the possibility of an instrument 
failure (specifically, a failure of the artifical horizon). As a result, 
the frame labeled VACUUM SYSTEM MALFUNCTION was activated, since such a 
problem could cause a false reading on the artificial horizon. This 
frame instructed them to CHECK TO SEE WHETHER the associated EXPECTATIONS 
ARE MET producing their first query: Wat is the reading on the 

suction gauge? 

The remaining three subjects (Figures 3b and c) took a different 
approach. They assumed the plane was in a (nose-down)- DESCENT and started 
looking for possible causes of such a descent. They therefore activated 
the POWER LOSS frame, which instructed them to test for power loss, 
checking for expected readings on either the manifold pressure gauge or 
the tachometer. 
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Figure 4 summarizes the alternative questions considered by 
these seven subjects during their efforts to generate an initial query. 
Thus far, then, we have developed the following model: 

1. Frames represent prototypical states of nature. (Labels for 
ten such states have been identified.) 

2. A frame is activated either by data that has been collected or 
by another frame that has been activated (i.e., by some set of 
enabling events). A number of such enabling events have been 
identified for particular frames. 

3. Once activated, a frame can be used to generate a query in 
order to seek additional information. 

4. An activated frame can be used to answer the question: Why are 

you interested in that information? 

5. The Instructions for Use in frames are based on three lines 
of reasoning: 

a. If a frame is a valid representation of the state of 
nature, (e.g., nose-down DESCENT), then the expected 
readings on certain instruments (listed by that frame) 
should be present. To assess that frame’s 
validity, the pilot should ask for the readings on 
those instruments. 

b. If an instrument reading is inconsistent with the 
hypothesized state of nature (i.e., the activated frame), 
that instrument may be malfunctioning. 

c. If a frame is a valid representation of the state of 
nature, then something must have caused that state of 
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Figure 3-4. Alternate Paths Leading to Initial Queries for Seven Subjects Activating the (Nose-Down) DESCENT Frame. 
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nature to occur. Assume the frame is valid and look 


for possible causes. 

6. Subjects differ in terms of: 

a. The instructions contained in particular frames. 

b. The slot-fillers within a frame that are (first) acted 

on when following an instruction (check manifold pressure 
for expected reading vs. check tachometer). 

In addition, the frame system developed thus far indicates that subjects 

may not use all of the available cues or data in order to activate an 

initial frame. The scenario that was read stated twice in a row: 

"The instruments indicate an increase in airspeed, 
a steadily decreasing altitude, and zero pitch." 

Yet the evidence reviewed above indicates that these pilots initially 

focused their attention on the indicated increase in airspeed and 

decrease in altitude. These two cues are both indicative of a nose-down 

DESCENT. The third instrument indication (zero pitch) is inconsistent 

with this hypothesis. (If the plane is descending with increasing 

airspeed, the nose should be pitch down. If the pitch is zero and the 

airspeed is increasing, the plane should not show a decrease in 

altitude . ) 

What, then, were the determinants of attention for these subjects? 

What made the airspeed and altitude information more salient initially? 

Three possible causes come to mind: 

1. Studies of human perception and attention suggest that: 

"the perceptual system actively attempts to 
reconstruct the external environment in an effort to cope 
with the massive volume of information it continually 
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encounters. ... the ’match-mismatch' notion clearly 
identifies the unexpected as a, if not the, crucial 
determinant of attention" (Dember and Warm, 1979, 
p. 131). 

Extending the same concept to the "perception" or comprehension 
of text (the scenario), it is predictable that, in this 
problem, the indications regarding airspeed and altitude should 
be more salient than that of pitch. Prior to hearing about 
these instruments indications, the subject was told that the 
plane has been cruising for 15 minutes at a constant altitude. 
Thus, a mental model or reconstruction of the situation would 
indicate a constant airspeed, no change in altitude, and zero 
pitch. This means that two of the cues, an increase in 
airspeed and a decrease in altitude are unexpected and hence 
predicted to be highly salient. The third cue, zero pitch, is 
consistent with the constructed mental model, and therefore not 
as likely to attract attention. 

2. Studies of human word recognition suggest that specific 

feature detectors (letters, shapes, etc.) are activated when a 
word is presented and that these feature detectors in turn 
activate word detectors (Morton, 1970; Rummelhart and Siple, 
1974). Meyer, Schvaneveldt , and Ruddy (1975) have suggested 
that when a word detector is activated , it may prime its 
associated feature detectors. The net result is that if a new 
word with some of the same feature detectors is presented, 
these feature detectors will fire faster than usual (i.e.. 
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these features will be noticed more rapidly, will be more 
salient) and the new word will be identified sooner. 

The same type of process could occur during the reading of 
the scenario.. The pi-lot has been told' that a problem exists 
and that he must diagnose its cause. It is quite plausible 
that, before listening to the scenario, the pilot speculates on 
possible problems. An unplanned descent is a common problem 
(relatively speaking) that is also very serious. Hence, it is 
a problem that the pilot is likely to conjecture about prior to 
hearing the scenario. Such a conjecture about DESCENT could 
prime associated triggers (such as decreasing altitude), 
increasing their salience. 

3. Bower, Black and Turner (1979) state that: 

“according to schema theory the understander must commit 
himself to some initial schema in order to understand 
sentences; yet the most diagnostic information may not 
appear in the text until later. That is, one can be lead 
down 'garden path’ stories” (p. 340). 

Given the predicted salience of the increasing airspeed and 

decreasing altitude (the unexpected events) and the fact that 

these two cues are presented first, the pilot may already have 

been in a DESCENT down the "garden path" before hearing about 

the zero pitch. 

If the activated frame (DESCENT) instructed him to 
consider the reading on the vertical speed indicator (Subjects 
#2 and # 10) or to consider a possible power loss (Subjects #4, 
#6, and #18), the information about zero pitch might 
easily have been ignored as irrelevant. 
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If, on the other hand, the activated frame instructed the pilot 
to consider the reading on the artificial horizon (Subjects #1 
and #5), the salience of the third instrument indication, zero 
pitch, would be increased and the cue would likely be noticed. 

In order to avoid information overload, then, the pilots may have 
used these types of mechanisms to focus attention selectively on some 
subset of the cues available in the scenario. 

Nose-Level DESCENT . The seven pilots discussed above reported 
that they thought the plane was in a nose-down descent. Subject #16, 
on the other hand, thought the plane was in a: 

"straight and level descent, nose on the horizon" 

(Task 4) . 

He proceeded to check for possible causes of descent after activating- 
the frame for Nose-level DESCENT (see Figure 3-5). 

Subject #13 also activated the DESCENT frame as evidenced by his 
comment that "we are basically in a situation where we are losing 
altitude." The evidence indicates that he, like Subject #16, activated a 
frame representing Nose-Level DESCENT. When asked about his impression 
of the plane's physical orientation (Task 4) he stated that initially he 
thought : 
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Frame Label: POWER LOSS 1 
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Causes: POWER LOSS 


Expectations: CHANGE IN TACHOMETER .\ 


READING 

Instructions: CHECK TO SEE 


WHETHER CAUSE 


Instructions: CHECK TO SEE WHETHER 

IS PRESENT 

• 

EXPECTATIONS ARE MET 

•r 


Supporting Comments: "Start checking the power instruments . 

0 > What does the tachometer show?" 

to 


*This label is supported by the data from Task 4 (see text). 


Figure 3-5. Subject #16 





"the plane was descending in a straight and level 
cruise situation but that I'm losing altitude. Nose on the 
the horizon." 

This recollection is consistent with his statements at the beginnning of 
the problem solving task: 


S#13: "What is the manifold pressure?" 

Experimenter: "Why are you interested in that?” 

S#13: "Very definitely we have a situation where we seem to be 

losing power. The fact that we're decreasing in altitude 
and our airspeed is remaining steady indicates to me that 
we are basically in a situation where we are losing 
altitude. I would expect the airspeed to stay fairly 
cons tant if we ' re coming down . " 

These three conditions (losing power, decreasing altitude and constant 
airspeed) are consistent with the behavior of a plane that is in a 
nose-level descent due to a loss of power. 

These data would suggest, then, that this pilot activated a frame 
for Nose-Level DESCENT by attending to two cues: a decrease shown on the 

altimeter and zero pitch shown on the artificial horizon. Having 
activated this frame, an associated default value (constant airspeed) was 
also activated, in spite of the fact that it contradicted information 
given in the scenario (see Figure 3-6). 

What is even more interesting is the fact that immediately before he 
asked for the reading on the manifold pressure gauge (see the quotation 
above). Subject #13 said: 
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Figure 3-6. Activation 


of Initial Frames for Subject #13 





"Let me get this straight now. Increasing airspeed, decreasing 
altitude and you mean pitch as far as being above or below the 
horizon based on the attitude [artificial] horizon." [This was a 
statement, not a question. No response was given.] 

Thus, this pilot heard all three cues initially. Then, paying 
attention to the pitch and altitude information, he activated the 
Nose-Level DESCENT frame. He also activated this frame's default value 
for the reading on the airspeed indicator (constant airspeed), distorting 
his memory. He believed (stated) at this point that the scenario 
indicated a constant airspeed. All of this occurred in a time period of 
less than one minute. 

Having activated the Nose-Level DESCENT frame, Subject #13 then 
followed its instructions and checked for a possible cause of descent. 
This resulted in the activation of the POWER LOSS frame, which generated 
his first query: "What is the manifold pressure?" (See Figure 3-6.) 

Subject #11 also initially activated the Nose-level DESCENT 
frame. His recollection (Task 4) that: 


"my first impression was level descent” 
is consistent with his statements immediately after hearing the 
scenario : 


"The first thing he should do is check his power. ... If power has 
not changed, then in level flight his airspeed wouldn't change. To 
make sure I'm right on the scenario, what does the airspeed indictor 
show? " 


Unlike Subject #13, this pilot did not distort his memory when the 
Nose— Level DESCENT frame was activated. He noted the possible 
inconsistency between the expected airspeed and his recall of the 
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scenario-given airspeed, and checked to make sure his recall was correct 
before continuing (see Figure 3-7). (After learning that the airspeed was 
in fact increasing he rejected the Nose-Level DESCENT frame in favor of 
the Nose-Down DESCENT frame.) 

MEMORY ERROR . Subject #9 also activated a MEMORY ERROR frame. He 
asked : 

"You say he's decreasing airspeed?" 

How he distorted his recall (the indicated airspeed was increasing) or 
how he arrived at the MEMORY ERROR frame cannot be determined from his 
data. After asking the above question and correcting his memory 
error, he activated the Nose-Down DESCENT frame. 
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■BITOT- STATIC SYSTEM MALFUNCTION'. The subjects discussed above 


activated a DESCENT frame. The seven subjects to be discussed next, 
however, initially focused their attention on the possibility of a 
PITOT-STATIC SYSTEM MALFUNCTION. 

Figure 3-8(a-e) shows the frame system constituents supported by the 
data from these seven subjects. ‘Again, comments are provided to indicate 
the data supporting the proposed model. 

The evidence suggests that these subjects activated the PITOT-STATIC 
SYSTEM MALFUNCTION frame because they noticed an unexpected change in the 
readings on two pitot-static system instruments, the altimeter and the 
airspeed indicator. Their attention appeared to focus on the fact that 
there was an unexpected change. They did not appear to consider whether 
the stated changes were actually consistent with a pitot-static system 
malfunction (they were not). 

There is again evidence of memory distortions once the PITOT STATIC 
SYSTEM MALFUNCTION frame was activated: 


"To be honest with you, at first I thought you said the airspeed was 
decreasing and the altitude was showing a decrease, you know, a loss 
of altitude. So I was thinking that those were exactly backwards." 

"When we first started the problem, I misunderstood the 
indications... I thought you said the opposite, like the airspeed 
was increasing, and the altitude was increasing. That’s what made 
me go to the static source at first" (data from Task 4). 
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Supporting Comments: 

"The airspeed and the altimeter are both part of the pitot-static system and if he's 
been using a constant power and a constant attitude, power plus attitude is equal to 
performance so there should have been no change in performance. Is the pitot heat on?” 
[Why? | "I was wondering if there might be in that area icing. Picking up some icing." 


Figure 3-8a. Subject #14 
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Supporting Comments: 

"My first thoughts are that there's definitely going to be a problem with the pitot-static system. 
What's the outside air temperature reading?" (Answer given: 37°F.) "There's still a possibility 

of icing build-up so pare of the way to determine that would be to flip on the pitot heat." 
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Figures 3-8b. Subject #20 
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Supporting Comments: 

SII3 - "Steadily decreasing altitude. Then I would also 

assume that includes then a showing a descent on the 
vertical speed indicator having a reading consistent 
with the altimeter. To try to narrow down is it a 
pitot-static system problem." 

S//19 - "Number one, look at the vertical speed indicator. 

It appears that it could be a problem with the 
pitot-static system." 


Figure 3-8c. Subjects # 3 and #19 
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Supporting Comments: 

"My first thing that I would think of would be something to do with either the static port 
or the pitot tube since those instruments, the airspeed and the static port are different, 
so 1 would think, also, the altimeter 1 know is getting its reading from the static port 
so the first thing I would think of would be a problem with the static port. What is the 
outside air temperature?" [Why? 1 "'Cause I'm thinking it could be a possibility of something 
freezing." 


Figure 3~8d. Subject //8. 
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Supporting Comments: 

S//12 - "What happens if I open the alternate static source?" (Why?) “I suspect from 
what is happening, since there is no pitch change, that there may be a blockage 
in the system chat's causing faulty instrument Readings. X don't know yet what 
but that's my first indication." 


Sltn - “What if I open my alternate static source?" (Why?) "To 6ee if X have a clogged port." 


Figure 3-8e. Subjects #12 and #17. 


*To be consistent with subject II 8 (see Figure 8d) the frame for BLOCKED STATIC PORT should be 
accessed through another frame ( PITOT-STATIC SYSTEM MALFUNCTION). 

Figure 8(a-e). Activation of Initial Frames for Seven Subjects Focusing on a Pitot-Static 
System Malfunction. 
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Thus, because they had activated the PITOT-STATIC SYSTEM MALFUNCTION 
frame, the pilots reconstructed their memories (Zechmeister and Nyberg, 
1982) to be consistent with such a state of nature. 

Figure 3-9 summarizes the alternative mental processes exhibited 
by these seven pilots while selecting their initial queries. 

ICING. Like three of the pilots who activated the PITOT-STATIC 
SYSTEM MALFUNCTION frame. Subject #7 thought that icing was causing the 
problem. We cannot tell, however, whether he was concerned with icing of 
some particular part of the plane: 

"You're in IFR conditions, so was there any visible moisture 
outside?" [Why?] "Because it might be in regards to icing." 

\ 

AIRSPEED INDICATOR MALFUNCTION . Subject #15 asked: 

"Is there an increase in noise outside the plane?" [Why?] 

"That would indicate an increase in airspeed, would back up that 
instrument indication." 

These comments make it clear that the pilot was attending to the 
indicated increase in airspeed, but provide little other insights into 
his mental processes. 

/ 

Summary. The goal of this subsection on the Initial Activation of 
Frames was to describe the mental processes that generated pilots' first 
queries. If we look at the information requests above, we see one 
"stimulus" (the scenario plus associated instructions) resulting in ten 
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different initial responses (first queries) from a total of twenty 
subjects (see Table 3-4). Two general questions arise: 

1. Why does a particular subject request a given piece 
of information? 

2. Why do not all subjects ask for the same piece of information? 

The frame system model outlined thus far indicates that there is a 
substantial amount of information processing that occurs before a pilot 
makes his first query. Individuals differ in the cues to which they 
attend while listening to the scenario, in the lines of reasoning that 
they apply once a frame has been activated, and in the slot-fillers 
contained in a given frame (Expectations: DESCENT INDICATED ON VERTICAL 

SPEED INDICATOR vs. Expectations: PITCH DOWN ON ARTIFICIAL HORIZON). 

They may also distort their memory to be consistent with the activated 
frame. These differences can account for the great heterogeneity found 
in the initial queries. 

Subjects activating the Nose-Down DESCENT frame (see Figure 3-3) 
were attending to two cues, an indicated increase in airspeed and 
decrease in altitude. Those activating the PITOT-STATIC SYSTEM 
MALFUNCTION frame (see Figure 3-8), on the other hand, appeared to focus 
on the information that there was some unexpected change in the readings 
on the airspeed .indicator and the altimeter. They did not attend to the 
specific directions of these changes when activating this MALFUNCTION 
frame. Such differences in cue selection led to radically different 
questions, ranging from queries about a POWER LOSS to tests for a BLOCKED 
STATIC PORT. 


75 



*vj 

CS 


Change 

on 

Airspeed 

Indicator 


Change 

on 

Altimeter 


Zero Fitch 
on 

Artificial 

Horizon 



Plcot-stadlcf 
System 1 


Ad Oat Firfi Query 


Blocked 


Pitot 

s Pitot Heat 

Pitot 


Tube 

7 Off? 

Tube? 


Icing? 

\ 


Blocked j 


Static 

Static 

r > 

L 

Port 

Port? 

i 

k 

Icing? 



Low Reading on 
Outside Air 
Temperature Gauge? 


Instrument Readings 
Return to Normal If 
Alternate Static 
Source Opened? 


Inconsistent 
Reading on 
Vertical Speed 
* Indicator? 


Figure 3-9. Alternative Paths Leading to Initial Queries for 
Seven Subjects Activating the PITOT-STATIC SYSTEM 
MALFUNCTION Frame. 
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Even when two pilots activated the same initial frame (such as 
Nose-Down DESCENT) they sometimes applied different heuristics to further 
evaluate that frame. Subjects #2 and #10 activated the DESCENT frame and 
tested its validity by looking at the vertical speed indicator for the 
expected reading. Subjects #4, #6, and #18 also activated the DESCENT 
frame, but checked for a possible cause of DESCENT, a POWER LOSS. Thus, 
the application of different "Instructions for Use" resulted in 
significantly different queries. 

Finally, even if two pilots activated the same frame and applied the 
same instructions, they still sometimes asked different questions. 
Subjects #6 and #18 were both concerned with POWER LOSS, but one checked 
the manifold pressure gauge for the expected reading while the other 
checked the tachometer. (The reading on the tachometer is actually 
uninformative since the Cherokee Arrow has a constant speed prop.) 

Organization of the Knowledge Structures 

The- previous analysis identified ten different frame labels, based 
on the pilots' first queries and associated- statements. Applying the 
same form of analysis to the remainder of the verbal protocols, we find 
evidence for eight additional frames: 

1. ARTIFICIAL HORIZON MALFUNCTION 

"Because there could be a malfunction with the artificial 
horizon." 

"I'm thinking that the artificial horizon's not working.” 
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STRUCTURAL ICING (WINGS) 


"Is there ice on the wings?" [Why?] 

"It gives you more drag and it adds weight." 

"Is- there - ice, am I receiving ice on the wings?" 

3. TRIM WHEEL MISPOSITIONED 

"Let’s check trim to see if had disturbed it, although I should 
see something on the pitch attitude indicator if I did have a 
problem with trim. What is the position of the trim tab?" 

4. DOWNDRAFT 

"At 6000 feet you could have turbulence which would cause a 
problem with maintaining altitude . . . could indicate whether 
it’s turbulent air, downdrafts." 


5 . GEAR DOWN 

"If the landing gear's down for some reason..." 

"Is the gear extended?" 

6. FLAPS DOWN 

"Are the flaps down? If the plane is trimmed for cruise 
flight, clean, then if you induce any drag that could take care 
of your airspeed and start a descent for us." 


7. BANKED PLANE 

"What does the artificial ‘ horizon show?” [Why?] "Are we in a 
bank?" 


8. YOKE MISPOSITIONED 

"If the pilot has decreased backpressure, pushed the elevator 
forward. . .which causes the nose to go down... What is the yoke 
position currently?" 


78 



All of the queries made by the twenty pilots can be accounted for in 
terms of attempts to access or to test the validity of the eighteen 
frames that have been defined. Below is ah exhaustive mapping of the 
interrelationships among these frames and the queries they generated. 

This mapping is consistent with all of the data. Not all pilots that 
activated a given frame made all of the queries listed. Also, the 
ordering is arbitrary in the listings of slot-fillers. Note that there 
are seven high-level frames (indicated by Roman numerals) in this frame 
system. (V and VI may not be high-level frames, but the available 
evidence did not establish at which lower levels they would belong.) 


I. 


NOSE-DOWN DESCENT 
A. Causes: 


1 . POWER LOSS 


Queries : 

a. 

b. 

c. 

d. 

e. 


Check for Expected Readings From: 

Manifold Pressure Gauge 
Tachometer 
Oil Pressure Gauge 
Oil Temperature Gauge 

Effect of an Increase in Power Using the Throttle. 


2 . STRUCTURAL ICING 


Queries: Check for 

a. Visible Ice on Wings 

b. Visible Ice on Windshield 

c. Visible Ice on Temperature Probe 

d. Reading on Outside Air Temperature Gauge 

e. Visible Moisture in Air. 

3 . DOWNDRAFT 

Queries : Check 

a. Weather Reports 

b. Whether Air (Ride) Feels Rough 

c. Outside Air Temperature Gauge 

d. Mountainous Terrain. 

4. GEAR DOWN 

Queries : Check 

a. Gear Down Light. 

5. TRIM WHEEL MISPOSITIONED 

Queries : Check 

a. Trim Wheel Position 

b. Artificial Horizon. 
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6. YOKE MISPOSITIONED (DECREASED BACKPRESSURE) 

Causes : 

a. ARTIFICIAL HORIZON MALFUNCTION 
Queries : Check 

a. Yoke Position. 

7 . FLAPS DOWN 

Queries : Check 

a. Flap Switch Position 

b. (Visible) Position of Flap. 

8 . BANKED PLANE 

Causes : 

a. YOKE MISPOSITIONED (LEFT TURN) 

Causes : 

1. ARTIFICIAL HORIZON MALFUNCTION 
Expectations/ Queries : 

a. Bank shown on turn and bank indicator 

b. Bank shown on artificial horizon. 

B. Expectations/Queries: Check for 

1. Nose-down on artificial horizon 

2. Descent shown on altimeter 

3 . Descent shown on vertical speed indicator 
4., Increase shown on airspeed indicator 

5. If backpressure applied to yoke, instrument indications 
of descent will cease or be reduced in magnitude. 

II. NOSE-LEVEL DESCENT 

A. Causes: 

1. POWER LOSS 

Queries: Check for Expected Readings From: 

a. Manifold Pressure Gauge 

b. Tachometer 

c. Oil Pressure Gauge 

d. Oil Temperature Gauge. 

2. STRUCTURAL ICING 

Queries : Check for 

a. Visible Ice on Wings 

b . Reading on Outside Air Temperature Gauge . 

3. GEAR DOWN 

Queries: Check 

a. Gear Down Light. 

4. FLAPS DOWN 

Queries : Check 

a. Flap Switch Position. 

5 . BANKED PLANE 

Queries : Check 

a. Artificial Horizon. 

B. Expect at ions /Queries: Check for 

1. Zero pitch on artificial horizon 

2. Descent on altimeter 

3. Descent on vertical speed indicator 
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4. Constant airspeed 

5. If backpressure applied to yoke, instrument 
indications of descent will cease or be 
reduced in magnitude. 


III. PITOT-STATIC SYSTEM MALFUNCTION 

A. Causes: 

1. BLOCKED STATIC PORT 

a. Causes: 1. ICED STATIC PORT 

Expectations /Queries: Check 

a. Outside air temperature 

gauge . 

b. Expectations /Queries: Check 

1. To see if instrument readings return to normal 
after alternate static source is opened. 

2. BLOCKED PITOT TUBE 

a. Causes: 1. PITOT TUBE ICING 

Expe ctations /Queries : Check 

a- To see if pitot heat is off 

b. Outside air temperature 

gauge . 

B. Expe ctations /Queries : Check 

1. To see if the readings on the altimeter, vertical 
speed indicator and airspeed indicator are 
inconsistent with one another. 


IV. ARTIFICIAL HORIZON MALFUNCTION 

A. Causes 

1. VACUUM SYSTEM MALFUNCTION 

a. Expectations /Queries: Check for 

1 . Low reading on suction gauge 

2. (Possible) Inconsistent readings on the 
artificial horizon and directional gyro as 
compared with the turn and bank indicator, 
magnetic compass and course deviation indicator. 

B. Expectations /Queries : Check 

1. For the effect of a left turn input (with yoke) 
on the artificial horizon. 

V. AIRSPEED INDICATOR MALFUNCTION 

A. Expectations/Queries: Check 

1. For an Increase in air stream noise outside cockpit. 

VI . ICING 

A. Expectations /Queries : Check for 

1. Visible moisture in the air 

2. Visible ice on the wings 

3. Visible ice on the temperature probe. 

VII. MEMORY ERROR 

A. Expectations /Queries : Check the accuracy of recall for: 

1. The airspeed indicator. 
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It is clear that such a frame system could be expanded to include 
more levels (POWER LOSS is caused by FUEL STARVATION, INDUCTION ICING, 
etc.) and additional hierarchies. These additional frames are 
unnecessary, however, to account for the data from this experiment. 

Each frame contains two slots, one for causes of that state of 
nature and one for expected instrument readings and observable 
conditions (visible ice on wings, etc.) if that state of nature exists. 
Each of these slots contains one or more entries (a list of possible 
causes or expected readings). Figure 3“10 summarizes this basic frame 
structure. 
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Frame Label: 


Causes: 

\Nnraea of 


jother frames 

Expectations : 

'l Observable 

* ■ . ... . ... • 

iData 

Instructions for Use: 



Figure 3-10. Frame Structure 
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Searching for Goal-States 


In this experiment, the pilots were told that: 

1 . There is a problem. 

2. The goal is to determine the cause of the problem. 

In terms of the proposed knowledge structures, this goal can be 
translated as follows: Complete the sentence: 

1. The problem is X; 

2. The cause of the problem is Y; 

where X is a Frame Label belonging to some "Problem" set: 

Problem = [NOSE-DOWN-DESCENT, 

POWER LOSS, etc.] 

and Y is one of the slot-fillers for the "Causes" slot in the frame with 
label X. 

Fault diagnosis can be described as a task of: 

1. Hypothesizing the presence of a particular problem 
(e.g., NOSE-DOWN DESCENT). 

2. Deciding whether this problem exists 

(e.g., checking the vertical speed indicator for a descent, 
etc .) . 

3. Determining the cause of this problem (e.g., checking for 
POWER LOSS, DOWNDRAFT, etc.). 

Because the problem may actually be a causally-connected chain of events-, 
it may be necessary to repeat this task in a recursive fashion until the 
initiating cause is discovered. 
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This problem-solving process can be illustrated by looking at the 
data for Subject if 1. (See the beginning of the Results and Discussion 
section for his verbal protocol.) 

Step 1 . 

Hypothesized Problem: Unexpected decrease in altitude and increase 

in airspeed. 

Possible Cause: NOSE-DOWN DESCENT 

Step 2. 

Hypothesized Problem: NOSE-DOWN DESCENT 

Test of hypothesis: Check for pitch down on artificial horizon 

Conclusions : Either the plane is not in a NOSE-DOWN DESCENT or 

there is an ARTIFICIAL HORIZON MALFUNCTION 


Step 3 . 

Hypothesized Problem: ARTIFICIAL HORIZON MALFUNCTION 

Possible Cause: VACUUM SYSTEM MALFUNCTION 

Step 4 . 

Hypothesized Problem: VACUUM SYSTEM MALFUNCTION 

Test of hypothesis: Check for low reading on suction gauge. 
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Subject #l's conclusion was that: 

"You have a nose-down attitude and the vacuum pump’s gone." 

Although no direct evidence is available to substantiate it, it is 
•plausible that this statement implies the activation or development of a 
script (Schank and Riesbeck, 1981) describing the following set of 
events: 

VACUUM SYSTEM FAILURE 
caused 

ARTIFICIAL HORIZON MALFUNCTION 
caused 

YOKE MISPOSITIONED (DECREASED BACKPRESSURE) 
caused 

NOSE-DOWN DESCENT. 

Fault diagnosis then, can be described as a process of recursively 
identifying problems and their causes until the person decides he has 
found the initiating cause. The data indicate that, in order to drive 
this process, subjects attempt to answer six types of questions: 

1. Is the currently activated frame a goal -state? (Have I 
achieved the goal of finding the (initiating) problem X and its 
cause Y?) 

2. What is the cause of the state of nature represented by this 
frame? 

3. Is this frame a valid representation of the state of nature? 

Are the expected instrument readings and conditions present? 
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4. If the currently activated frame has been rejected as a 
possible state of nature, can I find another frame to 
activate? 

5. Is there a recall (memory) error? 

6. Is there an Instrument malfunction? 

The performances of the twenty pilots in terms of pursuing these 
fundamental questions (asking about causes, expectancies, etc.) are 
summarized in Figures 3-11 to 3-15. A path from a high-level frame 
(e.g., NOSE-DOWN DESCENT) to a node labeled ’’Expectations" means that the 
pilot checked for presence of one or more of the instrument readings and 
conditions predicted to be present by that frame (e.g., DESCENT SHOWN ON 
VERTICAL SPEED INDICATOR) . A path from a high-level frame to a node 
labeled "Causes" indicates that the pilot tested for the presence of one 
or more of the possible causes associated with that frame. A path to a 
new high-level frame means that that frame was activated (and the 
originating frame de-activated) . Paths leading from "Expectations" or 
"Causes" back to a high-level frame indicate that the pilot proceeded to 
ask another one of the six questions listed above with respect to the 
same high-level frame. 

The number next to a link indicates the number of pilots who 
followed that path at least once (no pilot followed a path more than 
twice, and even this was unusual). The dotted arrows indicate special 
cases , where the pilots went directly from a violation of an expectation 
(the artificial horizon showed nose-level when it was expected to 
indicate nose-down) to a new frame (artificial horizon malfunction). 
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or from checking for possible causes directly to a conclusion. The only 
result not shown on these figures is the finding that Subject #7 
discovered the vacuum system failure, then checked for another possible 
failure in the pitot-static system (rejecting this possibility), finally 
returning to vacuum system failure as his conclusion. 

Note that Group A (Figure 3-11) consists of the eleven subjects who 
diagnosed the problem as a vacuum system failure. Group B (Figure 3-12) 
pilots concluded that there was an artificial horizon malfunction. Group 
C (Figure 3-13) concluded the problem was a downdraft. Group D pilots 
(Figure 3-14) concluded the cause of the problem could not be determined 
with the available information. 

Group E (Subject #20) discovered the presence of the artificial 
horizon malfunction. At that point he thought the plane was in a 
straight, nose-down descent that was not being indicated on the 
artificial horizon. He asked what would happen if he applied 
backpressure on the yoke to arrest the descent. When the expected 
response did not occur (because the plane was actually in a left spiral), 
he decided there must be some other problem. He failed to discover the 
left bank and concluded that he could not determine what the problem was 
(see Figure 3-15). 
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Figure 3-12. Processing by Subjects #12-15 (Group B) 
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Figure 3-13. Processing by Subject #16 (Group C) 
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Figure 3-14. Processing by Subjects #17-19 (Group D) . 
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Figure 3-15. Processing by Subject #20 (Group E) . 












Patterns of Performance 


The flow chart shown in Figures 3-11 to 3-15 is a high-level 
description of all the paths taken by one or more pilots. Groups A, B, C 
and D divide the subjects into four classes according to their final 
conclusions. (Group E is really a special case of Group B.) 

The most apparent differences among the groups are the contexts in 
which the six alternative questions (check for goal state, check for 
causes, test expectations, look for new frame, check for memory error, 
or consider instrument malfunction) are addressed. Group B, for 
instance, differs from Group A by the failure to check for possible 
causes of the artificial horizon malfunction. 

Cognitive Narrowing . The failure of Group B to determine whether a 
vacuum pump failure was causing the ARTIFICIAL HORIZON MALFUNCTION can be 
explained by a faulty ordering of the six questions in terms of 
application priority. Use of a simple rule would have almost certainly 
caused all the Group B pilots to discover the vacuum pump failure: 

Always check for possible causes of the state of nature represented 
by the currently activated frame before asking whether it is a 
goal-state. 

(During Task 3, all of the pilots in Group B demonstrated that they had 
knowledge of the relationship between the vacuum system and the 
functioning of the artificial horizon.) 

This explanation of Group B’s failure to seek a deeper cause is 
consistent with pilots’ explanations at the end of the experiment as to 
why they stopped without asking about the suction gauge: 
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"I just narrowed my vision down to one area, tunneled my vision 

down and stopped." 

Activation of Default Values . All three pilots in Group D activated 
the NOSE-DOWN DESCENT frame. They then proceeded to check for possible 
causes of the descent. When they failed to find a cause they stopped and 
concluded the cause of the problem could not be determined with the 
available information. 

Given the plane was in a nose-down descent, what accounts for this 
failure to find the cause? The answer lies in the activation of a 
default value. All three of these pilots reported that they thought the 
plane was in a straight nose-down descent (Task 4). Subject #18 even 
reported visualizing the turn and bank indicator, that the: 

"Turn and bank indicator showed straight and level." 

(In actuality he had been given no information about the turn and bank 
indicator, which showed a left bank.) 

This data suggests that the pilots in Group D activated a default 
value for the direction of the NOSE-DOWN DESCENT, that the descent was 
straight ahead. They did so in the absence of any data to support this 
assumption. (On the other hand the plane had been cruising straight 
ahead, and they had not received any information clearly indicating a 
turn.) 

The activation of this default value rules out the actual cause of 
the DESCENT, a BANKED PLANE. None of the subjects in Group D considered 
this as a possible cause of the descent. 
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Similarly, none of the pilots in Groups A and B considered a BANKED 
PLANE as a possible cause of the NOSE-DOWN DESCENT when they started 
looking for such a cause (see Figures 3-11 and 3-12). These pilots also 
ass.umed the plane was -descending straight ahead when checking for a cause 
of the descent. The five subjects in Group A who ultimately concluded 
the plane was in a left-bank did so only after they discovered the vacuum 
system malfunction. 

Changing Focus . Like the pilots in Group D, many of the subjects in 
Groups A and B checked for possible causes of a NOSE-DOWN DESCENT (see 
Figures 3-11, 3-12, and 3-14). They all failed to find the cause, a 
BANKED PLANE, because of a faulty assumption. Yet the pilots in Groups A 
and B continued and discovered the ARTIFICIAL HORIZON MALFUNCTION, while 
those in Group D stopped and concluded the cause could not be determined. 

The difference in performance was that, after failing to find a 
cause for the NOSE-DOWN DECENT, the pilots in Groups A and B checked to 
see whether the "Expectations" for this frame were present. In 
particular, they checked to see whether the artificial horizon showed 
nose-down. When they found it did not, they began to investigate a 
possible ARTIFICIAL HORIZON MALFUNCTION. 

Why did some pilots (Groups A and B) think about expected 
instrument readings in this context (failing to find a cause of DESCENT) . 
while others (Group D) did not? One possible answer lies in the 
functions that checking expectations could serve. Addressing this 
question would help the pilot discover that he has: 


96 



1. Activated the NOSE-DOWN DESCENT frame based on insufficient 
information (such as increasing airspeed alone). 

2. Activated the NOSE-DOWN DESCENT frame based on faulty 
information due to a recall error (memory distortion). 

3. - Activated this frame based on faulty information due to an 

instrument malfunction. 

4. Activated this frame based on faulty assumptions (activation of 
incorrect default values). 

Thus, asking this question could help the pilot discover that he is 
focusing his attention on the wrong frame(s). 

Unlike the subjects in Groups A and B, the pilots in Group D never 
considered the possibility that they were addressing the wrong question. 
They assumed that the state of nature was a straight NOSE— DOW DESCENT 
and tried to determine the cause. Subject #18 for instance, checked for: 
POWER LOSS 
STRUCTURAL ICING 
GEAR DOWN 
FLAPS DOWN 
DOWNDRAFT 

Upon finding that none of these causes were present, he immediately 
concluded that he could not diagnose the problem. 
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Slot-Fillers . If Subject #18 and the other pilots in Group D had 
switched their attention from the "Causes" slot of the NOSE-DOWN DESCENT 
frame to the "Expectations" slot, this could have focused their attention 
on the inconsistent reading on the artificial horizon and, consequently, 
on the ARTIFICIAL HORIZON MALFUNCTION. This assumes, of course, that the 
pilots looked at the right slot-filler in the "Expectations" slot. 

In general the "Causes" and "Expectations" slots in a frame have 
multiple slot-fillers. When a question is addressed by focusing on one 
of these slots , it can be asked with respect to one or more of the 
slot-fillers. If it is not applied to the right slot-filler, critical 
information may be missed. 

In this problem, five slot-fillers were identified for the 
"Expectations" slot in the NOSE-DOWN DESCENT frame. In terms of solving 
this particular problem the critical expectation is that of "Nose-down on 
artificial horizon." Two pilots in Group A activated the NOSE-DOWN 
DESCENT frame, and then checked the "Expectations" slot. The slot-filler 
they checked was "Descent shown on vertical speed indicator." When that 
expectation was met, they concluded that the correct frame had been 
activated and looked at the other slot (checked for possible causes) 
immediately, rather than considering all of the slot-fillers in the 
"Expectations" slot of the NOSE-DOWN DESCENT frame. Had they not tested 
additional expectations upon failing to ■ identify a cause, they would not 
have solved the problem. 

This suggests that another general rule should be followed: 
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Check all of a frame's "Expectations" slot-fillers before looking 
for possible "Causes". Otherwise, the pilot may fail to detect 
memory and activation errors, instrument malfunctions, and faulty 
(default) assumptions. 

Cross-Checking Instruments . The proposed fault diagnosis process 
relies very heavily on the contents of the frames in order to achieve 
various goals. If those contents are inadequate, then checking all of 
the slot-fillers in the "Expectations" and "Causes"^ slots may not be 
sufficient to ensure detection of instrument malfunctions, incorrect 
assumptions, etc. Subject #16 (Figure 3-13) illustrates this point. 

This pilot was the only subject to consider a BANKED PLANE as a possible 
cause of DESCENT. He activated the NOSE-LEVEL DESCENT frame and checked 
for possible causes of DESCENT, including a BANKED PLANE. To check for a 
left bank, he asked for the indication on the artificial horizon (which 
showed straight and level flight). He did not check anything else to 
rule out a BANKED PLANE, and therefore falsely concluded that this was 
not the cause of the DESCENT. 

This suggests that the slot-fillers for frames must be selected so 
as to ensure detection of instrument malfunctions. In this example, the 
"Expectations" slot of the BANKED PLANE frame should have directed the 
pilot to check both the artificial horizon and the turn and bank 
indicator to avoid reliance on a single instrument (system). Similar 
consideration is needed in selecting slot-fillers to ensure detection of 
activation and memory errors and incorrect activation of default values. 
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Selecting the Right Question . In an earlier section, the 


determinants of attention to certain cues were discussed. A similar 
issue can be raised with respect to the six fundamental questions that 
pilots asked': 

How does a pilot decide which question to ask in a given context? 

It has been suggested that these questions are addressed in order to 
achieve certain objectives (in which case we could ask how objectives are 
selected). It has been implied, for instance, that a pilot tests for the 
presence of a frame’s expectations in order to discover: 

1. Activation of the wrong frame based on incomplete information. 

2. Memory distortions. 

3. Instrument malfunctions. 

4. Activation of incorrect (default) assumptions. 

Thus, asking a question about "Expectations” represents a plan (Wilensky, 
1983) for achieving an objective such as finding a new frame upon which 
to focus attention. This type of question selection processs can be 
modeled as a production system, as illustrated by the following example. 

If none of the possible causes associated with a frame are present, 

check to see whether the wrong frame has been activated (based on 

incomplete or erroneous information) . 

To see whether the wrong frame has been activated, check for the 

presence of that frame’s "Expectations.” 

Pilots activating the PITOT-STATIC SYSTEM MALFUNCTION frame might 
check for two possible causes, PITOT TUBE ICING or STATIC PORT ICING, by 
checking the outside air temperature gauge. These would be rejected as 
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possible causes because the temperature is too high for icing. The pilot 
should then check the "Expectations" associated with a PITOT-STATIC 
SYSTEM MALFUNCTION. Finding no inconsistencies on the airspeed 
indicator, altimeter and vertical speed indicator, the pilot would reject 
this MALFUNCTION frame. 

Goal -Directed vs. Data-Driven Processing . The previous subsection 
suggested that information such as "Expectations" is used in order to 
achieve an objective. The implication is that the objective is selected 
by the pilot and that this directs the selection of an appropriate plan, 
which then focuses on a particular slot. 

An alternative (or additional) explanation is that focusing 
attention on a particular slot , like the initial activation of some 
frame, is a data-driven process. Thus, a pilot might consider whether 
the reading on the artificial horizon is consistent with a NOSE DOWN 
DESCENT not because he feels it will help him achieve some objective, but 
because : 

1. He remembers getting information about the plane's pitch. 

2. Pitch information is relevant to one -of the "Expectations" 
slot-fillers in the NOSE-DOWN DECENT frame. 

This type of data-driven processing could be accomplished by some type of 
intersection search (Anderson and Bower, 1980) in which data activates 
both frames and slot-fillers. Questions are generated by looking at 
the activated slot-filler within the currently activated frame. 
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Such a data-driven process can still achieve objectives. Its 
success in doing so, however, depends upon the way in which the knowledge 
structures are organized and activated. 

This concept of data-driven selection of questions offers an 
alternative explanation for the failure of the Group D pilots to discover 
the artificial horizon malfunction. Previously, it was suggested that 
these pilots failed to check the "Expectations" slot of the NOSE-DOWN 
DESCENT frame because they failed to identify appropriate objectives 
(e.g., discover instrument malfunctions or activation of incorrect 
default assumptions). The proposed data-driven processor, however, would 
account for this failure in terms of an attentional error: The pilot 

failed to attend to the indication of zero pitch in the scenario. As a 
result there was no activation of instruction to check the artifical 
horizon, which was contained as a slot-filler in the "Expectations'’ slot 
of the NOSE-DOWN DESCENT frame. 

This explanation is consistent with the fact that none of the pilots 
in Group D recalled the indication of zero pitch in Task 2. (All of the 
other pilots in the experiment recalled this indication.) 

A Normative Model . 

Based on a comparison of the processing strategies of successful and 
unsuccessful pilots on this task, recommendations for a high-level 
control structure can be made. Since only one problem has been studied, 
however, it is impossible to determine how widely applicable these 
recommendations are. 

This control structure must decide what questions to 
address in a given context. One alternative, of course, is to put 
explicit "Instructions for Use” in each frame as was done earlier 
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in this paper. To the extent that the same questions are to he asked 
repeatedly, however, this becomes a less desirable approach. 

The proposed control structure consists of an ordered list of 
objectives, along with a set of plans designed to attain these 
objectives. The plans act upon the contents of various knowledge-bases 
and a working memory (see Figure 3-16). 

The first knowledge-base contains the eighteen frames identified 
earlier. The second contains objectives and associated plans for 
attaining them. The third contains the pilot's world knowledge and 
recall of past experiences related to aviation (episodic memory). The 
working memory serves to store the pilot's (episodic) memories of the 
scenario, the sequence of frames he has activated and objectives he has 
achieved, the information he has requested, the currently activated 
frame, etc. 

The pilot's objectives, in order of priority, are: 

I. Activate a frame representing the plane's physical activity and 
orientation (e.g., NOSE-DOWN or NOSE-LEVEL DESCENT), This is 
primarily a data-driven process that occurs as the pilot 
listens to the scenario. The control structure can influence 
this process by priming certain cues, as discussed earlier. 

. This priming involves accessing general aviation knowledge 

concerning common problems, and then activating the relevant 
frames. This would be done before listening to the scenario 
(or possibly while listening to it). 
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Figure 3-16 


System Architecture 


ORIGINAL 'PAGE si 
OF POOR QUALITY 






II. Make sure the correct frame has been activated. 

A. Ensure that the frame was not activated because of 
attention to the wrong cues. 

B. Be certain that the frame was not activated based on 
insufficient data. 

C. Make sure the frame was not activated based on incorrect 
data due to : 

1. a memory distortion. 

2. an instrument malfunction. 

D. Determine whether any default values have been activated 
incorrectly. 

This goal can be achieved by checking all of the "Expectations 
for the activated frame (assuming the right slot-fillers are 
present). Do not rely on the contents of working memory for 
these checks. It is important that the list of slot-fillers 
for the "Expectations" slot contain cross-checks for the 
various instruments .in order to ensure detection of instrument 
malfunctions . 

III. If an expectation is not met, either an instrument has 

malfunctioned, the wrong frame has been activated, or an 
incorrect default value has been activated. Check for an 
instrument malfunction first by activating the associated 
instrument malfunction frame. Objectives II and IV should be 
applied to use the contents of this malfunction frame. 
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If the instrument is malfunctioning, revise the contents of the 
working memory appropriately and return to Objective I. Use 
the contents of the working memory as cues for triggering a 
frame or replacing default values. Do the same thing if the 
instrument is not malfunctioning, since this implies the wrong 
frame or default value was activated originally. 

IV. Once it has been established that the correct frame has 

been activated, try to determine what is causing this state of 
nature. Activate the "Causes" slot for the frame, checking all 
of the slot-fillers until a cause is found or the list has been 
exhausted. Test each possible cause, activating the associated 
cause frame and testing its expectations. 

V. Repeat Steps II-IV in a recursive fashion, using as the 
activated frame the cause that has just been identified. 
Continue until no more causes in the chain of events can be 
identified . 

Those pilots who failed to fully diagnose the problem appeared to 
have the necessary frames at their disposal. Their high-level control 
processes, however, deviated from this normative model in significant 
ways . The organization of their processes failed to reflect the 
objectives of: 

1. Avoiding memory distortions. 

2. Detecting instrument malfunctions. 
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3 . Avoiding incorrect activation of a frame due to attentional 
errors and use of insufficient information. 

4. Avoiding false inferences due to activation of incorrect 
default values. 

5. Avoiding cognitive narrowing (failure to detect a causally- 
related chain of events). 

This normative model could be realized as an expert system (Barr and 
Feigenbaum, 1981; Davis, 1982; Nilsson, 1980; Nau, 1983). It is not 
clear, however, whether it is compatible with the cognitive processes 
observed in these pilots. If the attainment of these objectives is 
normally an implicit, data-driven process in human fault diagnosis, 
resulting from well-designed knowledge structures and attentional 
processes, then f amilizarizing pilots with such a control structure might 
have little impact on performance. 

Criticality of Assumptions 

Certain assumptions were made in order to permit the development of 
a parsimonious general model of performance. The primary effect of these 
assumptions was to model pilots as accessing all frames through the 
higher-level frames. It is possible, of course, that any frame at any 
level in the hierarchy can be activated directly by external cues. This 
experiment does not permit an evaluation of this alternative. 
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Representation of Knowledge 


A frame system using (verbal) symbol manipulations was used to model 
performance. It is possible that another representation using some 
analog process (a qualitative or quantitative simulation) as a mental 
model of flight could account for the same data. 

Extrapolation to a General Aviation Setting 

The objective of this study was to better understand the manner in 
which knowledge structures are used in fault diagnosis. The problem 
context was "similar" to a vacuum pump failure in a light plane. Can we 
make any inferences about performance in an actual aircraft? 

The major difficulty with making such an inference is the lack of 
similarity between our problem setting and an actual flight in terms of 
information input. The visual displays normally available to pilots 
provide a much different form of data-gathering . Our problem setting is 
much closer to that of a pilot conversing with an intelligent computer 
that is flying the plane. We might expect much more data-driven 
processing with the availability of the normal visual displays. 
Nevertheless, the knowledge structures and cognitive processes studied in 
this experiment probably play some role in actual on-board fault 
diagnosis. Thus, some of the concepts identified may provide insights 
into performance in a realistic setting. Some speculations are provided 
below: 
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1 


. Attention is attracted to unexpected events. If, in a real 

flight, the vacuum pump fails and our scenario becomes reality, 
the pilot's attention may not be attracted to the artificial 
horizon. Instead, it will probably be drawn to instruments 
that show an unexpected change. If that instrument is the turn 
and bank indicator, he will probably detect the artificial 
horizon malfunction quickly. On the other hand, if attention 
is drawn to the unexpected changes on the altimeter, vertical 
speed indicator or airspeed indicator, the pilot may be very 
slow to note that it is the failure of the artificial horizon 
that is causing his problems. Thus, from the standpoint of 
detecting vacuum system failures, the artificial horizon is a 
poorly designed instrument. Its failure to (always) give an 
unexpected change in reading when the vacuum system 
malfunctions could contribute to slow or even unsuccessful 
attempts to diagnose the problem. This design problem is 
exacerbated by the fact that the artificial horizon sometimes 
does tumble when the vacuum pump fails. This could generate an 
expectancy that would further reduce the pilot’s tendency to 
focus attention on the artificial horizon. 
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2 


Pilots behave as though activated default values are based on 
actual data (are not assumed values). If a pilot assumes he is 
flying straight because he is unaware of making any yoke 
movements (while tracking the failing artificial horizon) , he 
may not look at the turn indicators very quickly (or at all) . 

He may instead attend to the instruments relevant to the 
perceived problem (descent) . An activated default value in a 
frame may also influence other values in the same or other 
frames via some inheritance process. Thus, BANKED plane may 
not be considered as a possible cause of DESCENT because the 
pilot "knows" he's flying straight. 

3. Pilots tended to assume that their instruments were functioning 
properly, accepting this by default rather than by 
cross-checking the appropriate instruments. Thus, pilots - may 
activate a frame based on incomplete or inaccurate information 
(due to memory distortion or instrument malfunction). They may 
then assume it is the correct frame (e.g., DESCENT) and proceed 
to look for causes of the associated state of nature, instead 
of first making sure their understanding of the plane's 
activity is correct. This "cause chasing” may use up valuable 
time . 
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Future Issues 


A general model of fault diagnosis has been proposed. This model raises 
a number of issues about how people perform such tasks, and about how 
performance can be improved. These questions can only be addressed by 
designing additional experiments to collect converging and supplementary 
evidence. 

The next section describes alternative approaches to modeling fault 
diagnosis without benefit of verbal protocol analysis. The models discussed 
there are based upon PLATO® data which is nearly experimenter free. In that 
situation one must base all of his conclusion on observed behavior without 
benefit of discussion and/or rationalization about why particular information 
may or may not be important to a particular subject. The combination of frame 
models with more traditional graphical aids and stochastic process 
representations will hopefully provide a rich picture of how pilots respond to 
critical in-flight events. 
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IV. Modeling Instrument Scan Patterns 


During the course of this research project and its predecessors, a 
wealth of data on information search patterns was collected for over 100 
pilots. Each pilot/subject was asked to diagnose from one to five 
critical event scenarios. Some did this by paper and pencil techniques, 
some by PLATO computer graphics displays, and some by protocol analysis. 
This section describes modeling attempts directed toward discerning 
common solution strategies among subgroups of pilots. 

Order of Inquiry 

Early descriptive models for paper and pencil subjects were reported 
by Rockwell and Gif fin (1981). Subjects participating, in the PLATO® 

experiments were discussed by Rockwell and Giffin (1982). In both cases 

heavy emphasis was placed upon statistical analysis of data which 
attempted to relate diagnostic performance to pilot experience, knowledge 
and biographic information. Some broad generalizations about diagnostic 
behavior were made from these data, but no model which could predict how 
a given pilot might behave when confronted with a new scenario was 
obtained . 

One tool developed in this early research was the Pilot .Information 
Plot (PIP). These PIP charts permitted one to see graphically how any 
particular subject searched for information on any given scenario. The 
PIP charts offered a convenient way to conjecture the hypotheses and the 
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method of testing being pursued by a subject but were not helpful in 
discerning global strategies across -many subjects. Each trajectory 
appeared to be unique for that subject and that scenario. 

Figure 4-1 is a prototype PIP that one might conjecture would 
generated by a knowledgeable, efficient pilot. This trajectory was 
generated after studying the entire pool of subjects and is put forth as 
a subjective composite for "good" subjects. The scenario, which involves 
a vacuum pump failure, provided the subject with the following 
situational data: 

1) Increasing airspeed 

2) Decreasing altitude 

3) Level flight attitude indication 

4) Instrument meteorological conditions. 

From these data, the better subjects appeared to deduce the following: 

1) The instrument readings are contradictory. 

2) One or more of the instrument readings is in error. 

3) Two are static system instruments, one is a vacuum 
instrument. 

A confirmation of static system performance often involved checking 
the VSI and alternate static source. A confirmation of vacuum system 
performance might include checking vacuum instruments against independent 
information e.g. turn and bank or magnetic compass. Finally, the key 
element of suction gage reading offered confirmation of the vacuum pump 
failure hypothesis. 
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Figure 4-1. Sample PIP Chart 


114 



Frequency versus order of inquiry diagrams were also prepared for 
successful and unsuccessful groups of subjects in an attempt to isolate 
apparent hypothesis generation and preferred groupings or "tracks" of 
information inquiry. The clustering of inquiries looked remarkably 
similar across groups relative to incorrect hypotheses with the 
distinction between successful and unsuccessful subjects being mostly one 
of which group generated the proper hypothesis to test. These diagrams 
and their analysis appear in the 1982 report. 

Script Norms 

The concept of "script norms 1 ' as discussed by Bower, Black and 
Turner (1979) was modified for use in analyzing the entire group of PLATO 
subjects. Bower, Black and Turner' asked subjects to write a list of 
actions describing what people generally do in some common situaiton e.g. 
attend a lecture. The question was whether people agree in the actions- 
they mention. They tested this by examining the distributions of how 
many actions were mentioned by varying numbers of subjects. They 
designated the group's script to be those events mentioned by more than 
25 percent of the subjects. 

In the context of the CIFE scenarios, script norms are developed by 
tabulating distributions for the number of items of information requested 
by varying numbers of subjects. Here the pilot/subjects are being asked 
to recognize script relevant information from a large pool of potential 
information rather than being asked to generate a script from a 


115 



■ 


completely unstructured environment. The resulting distributions reveal 
both the level of agreement among subjects about which data are important 
and an indication of what groups of data belong together. 

Figures 4-2 through 4-6 are the script norms for the five scenarios 
administered to the computer aided testing (CAT) PLATO subjects. These 
graphs show the explicit items of information requested by different 
fractions of subjects. 

The general shape of the distributions is the same for all five 
scenarios. As was the case with Bower, Black and Turner, there appears 
to be a sharp difference at the 20 percent point for all scenarios except 
scenario five . Scenario five involved a broken muffler baffle leading to 
reduced power available and was considered by most subjects to be 
"unsolvable” . This caused many subjects to search the panel without any 
strong hypotheses about the problem diagnosis. Consequently the script 
norms for that scenario show many more items requested by a large 
portion of subjects as compared to the simpler scenarios. 

For purposes of this discussion we will arbitrarily define the 
group’s script to be those items requested by more than 40 percent of the 
subjects. Since scenario five was different from the rest and was 
administered to only a small number of subjects, it will not be 
considered in the rest of this discussion. 

Scenario one, an oil pressure gage leak, and scenario three, a 
magneto drive gear failure both concern obvious engine related symptoms . 
As expected, the group scripts for both scenarios are heavily laden 
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with engine health symptoms. The triumverant of oil temperature, 
cylinder head temperature and oil pressure was sought by nearly every 
subject in both scenarios. The key elements, fluid leaks for scenario 
one and left magneto for scenario three, were selected by at least 60 
percent of the subjects. The fact that these elements provided 
conclusive information for solution is reflected by the 52.4 percent and 
63.6 percent of the subjects who correctly diagnosed one and three 
respectively. 

Scenario two, the vacuum pump failure, and scenario four, the frozen 
static port were apparently much more perplexing. Only 23.8 percent of 
the subjects correctly identified the vacuum failure and only 43.2 
percent identified the frozen static port. In examining the script norms 
for those scenarios , it appears that ice is of prime concern to both 
groups. Alternate static questions, pitot heat, VSI and wing condition 
show up in both scripts. Since suction gage reading is not in the 40 
percent script cutoff for scenario two, it is obvious that the number who 
correctly solved that problem should be small. 

It is noted that pilots as a group seem to agree on the importance 
of a relatively small number of information items in problem diagnosis. 
The script for scenario one had only 7 items of information which were 
requested by over 40 percent of the subjects, scenario two had 8 items, 
scenario three had 13 items and scenario four had 11 items. These are 
only small percentages of the some 110 items of information which were 
available to the CAT subjects. 
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Figure 4-2. Oil Pressure Sage 
Scenario # 1 
29 CAT Subjects 
Script Norms 
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Figure 4-3. Vacuum Pump 
•v. Scenario $2 
42 CAT Subjects 
Script Norms 
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Figure 4-4. Magneto Failure 
Scenario #3 
36 CAT Subjects 
Script Norms 
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Figure 4-5. Frozen Static Port 
Scenario if 4 
37 CAT Subjects 
Script Norms 
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Figure 4-6. Broken Baffle 
Scenario 95 
14 CAT Subjects 
Script Norms 
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Distance Measures 


Another attempt to determine the existence of a common frame system 
for pilots involved calculation of the average distance between 
information items as a measure of closeness in memory structure. 

Clusters of information defined by short distances and high frequency of 
occurrences were sought as an alternative to the earlier ad hoc 
definitions of "tracks" used in the PIP charts. Distance is defined as 
the number of requests between any two items of information. 

For this analysis a consolidated state list containing 34 items was 
generated. Here closely related requests, e.g., yoke, yoke foreward and 
yoke back, were combined into a single state. The resulting list of 
states is shown in Table 4-1. 

Scenario two, vacuum pump failure, and scenario three, magneto drive 
gear failure, were analyzed across all computer aided testing subjects. 
Pairs considered significant were limited to those with an average 
distance measure of less than or equal to 3 as calculated from at least 3 
subjects. The pairs of items meeting these requirements for scenario two 
are listed in Table 4-2. Similar pairs for scenario three are listed in 
Table 4-3. 

By reading horizontally one can deter m ine the most likely neighbors 
for any given information inquiry. For example, in the vacuum pump 
scenario a request for outside air temperature (OAT) is likely to be 
followed by airspeed, altimeter, vertical speed, pitot heat and yoke. 
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Table 4-2. Vacuum Pump Scenario; Distances <_ 3 Inquiries; Number of Observations 
> 3 Subjects. 



















Table 4-3 . Magneto Gear Scenario; Distance <_ 3 Inquiries; Number of 
Observations > 3 Subjects. 



































Since OAT relates closely to an icing hypothesis one might conjecture 
that the rest of the members of this cluster indicate how this group of 
subjects might continue to test that hypothesis. 

By reading vertically one can determine from what elements subjects 
are likely to enter any given state. For example, in the vacuum pump 
scenario, inquiries about the altimeter may result from information about 
outside air temperature, airspeed, artificial horizon, manifold pressure 
and vertical speed. 

The vacuum pump scenario had the lowest successful diagnosis rate 
(23.8%) of the four main scenarios tested. The clusters shown in Table 
4-2 may therefore be reflecting a mix of hypotheses. Most subjects 
seemed to entertain an icing hypothesis at least sometime during their 
diagnosis, but jumped around somewhat when that could not be supported. 

The magneto drive failure scenario on the other hand had the highest 
success rate with 63.6% of the subjects solving it. Consequently the 
clusters in Table 4-3 may be more indicative of the true closeness in 
memory structure among items. For example, oil temperature is close to 
oil pressure, cylinder head temperature and ammeter readings all of which 
related directly to engine health. Similarly, requests for cylinder head 
temperature are likely to follow tank selector, oil pressure, oil 
temperature, throttle and exterior condition inquiries. Tank selector 
and exterior condition inquiries leading to CHT suggest a switch in 
tracks. The other requests are consistent with an engine health track of 
inquiry. 
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Markov Models 


The theory of Markov chains offers an intriguing model for 
diagnostic information gathering. It seems reasonable to ask whether or 
not there is a multi-stage dependence among inquiries, e.g. does the next 
inquiry depend upon the last 1, 2, n inquiries? The model to be 

examined is that of an .n^ order Markov chain in which a transition into 
state j on the k^ inquiry depends upon the states occupied on inquiries 
k-1 , k-2, k-n. When n = 1, the process is a common first order 

Markov chain and when n = 0 it is an independent process. A first order 
process has one state dependence expressed as 

PrtijJi!, i 2 , i k _i] = Pr [i k | i k _ X ] 

Independent trials on the other hand carry no historical information from 
inquiry to inquiry, expressed as 

Pr[ik|ii,i2,...,ik-l3 = Pr[i k ] 

Hypotheses concerning the order of Markov chains which might be- used 
to model information search trajectories were tested on five different 
groups of pilots on each of three scenarios. Scenario one was eliminated 
because not all subjects took it. The groups were selected from those 
which proved significantly different in the t-tests previously reported 
by Rockwell and Giffin (1982). The groups tested were 

1. All CAT (PLATO) subjects - 39 pilots 

2. Successful subjects with high correctness scores - 11 pilots 
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3. Non-private pilots - 22 pilots 

4. Private pilots - 16 pilots 

5. Subjects with high knowledge test scores - 10 pilots 
Three different sets of state definitions, representing various 
consolidations of inquiries were tested. These sets of states are listed 
in Table 4-4. 

* 

The hope was to find a homogeneous group of pilots and a properly 
descriptive set of state variables so that search trajectories could be 
combined in order to test hypotheses. Ideally, one pilot should be 
subjected to repeated exposure to the same decision-making problem in 
order to compute the frequencies with which he would move between 
inquiries. However, because of the nature of the scenarios given it was 
not reasonable to repeat trials for the same individual once he had been 
exposed to the true solution. For that reason the (tenuous) assumption 
of homogeniety among pilots within a group was made. The relative 
frequency of moving from state i to state j. was then calculated by 
pooling the number of inquiries across all pilots in a group who followed 
i with j and dividing by the total number of i-inquiries observed within 
the group. 

The statistical tests performed were those suggested by Anderson and 
Goodman (1957). They assumed repeated observations of the same chain and 
based their analysis on asymptotic distributions. Their approach for an 
N-state chain is to: 
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39-STATE SET 


1 . 

Breaker Panel 
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2. 

Suction 

2. 

Vertical Speed 

3. 

Outside Air Temp | 

3. 

Altimeter 

4. 

Airspeed 

4. 

Alternate Static 

5. 

Artificial Horizon 

5. 

Pitot Heat 

6. 

Altimeter 

6. 

Outside Air Temp. 

7. 

Vertical Speed 

7. 

Structural Ice 

8. 

Turn and Bank 

8. 

Air Traffic Control 

9. 

Directional Gyro 

9. 

Engine Gages 

10. ■ 

Pitot Heat 

10. 

Artificial Horizon 

11. 

Yoke 

11. 

Directional Gyro 

12. 

Alternate Static 

12. 

Suction 

13. 

Tachometer 

13. 

Yoke 

14. 

Manifold Pressure 

14. 

Turn and Bank 

15. 

Mixture 

15. 

Magnetic Compass 

16. 

Throttle 

16. 

Other 

17. 

Propeller 



18. 

Oil Temp. 



19. 

Oil Pressure 



20. 

Magneto 



21. 

Structural Ice 



22. 

Air Traffic Control 



23. 

Other 

i 



Table 4-4. State Definitions for Markov Tests 


ORIGINAL PAGE E$ 
OF POOR QUALITY 



1) obtain 1, 2, . .., k stage transition frequencies 

2) calculate transition probabilities for a total of 
n observations. 

4) compute a likelihood ratio statistic 

5) test the significance of the result. 

Transition probabilities are computed as follows: 



Total inquiries for element k 
Total inquiries 


p ■= = Number of inquiry sets (j,k) 

nj Total number of j inquiries 


■p _ _ Number of inquiry sets (i,j,k) 

ijk n^j ~ Total number of (i,j) inquiries 


The likelihood ratio is given by 
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N 

A = n 

i, j , . . .k, £-1 


p j,...k,a 




n. 


i, j j • • »k,£ 


The test is for a chain that is of order (r-1) against an 
alternative of r. That is 

Ho: Chain is of order (r-1) 

: Chain is of order r 

The test statistic - 2Zn X can be shown to be Chi square distributed 

2 _i 2 

(x ) with N r 1 (N-D degrees of freedom under the null hypothesis. 

The procedure is to begin with r=l and continue testing with increasing 
order of r until Hq cannot be rejected. At that point one can conclude 
that the chain is of order r. 

The above tests were applied to the previously described 
-combinations of subjects and scenarios. The results are noted in Tables 
4-5 through 4-9. Summary conclusions are contained in Table 4-10. 

If taken at face value these tests' offer some surprising 
conclusions. They suggest that for the groups of pilots tested, these 
scenarios can evoke no stronger dependence than that of a first order 
Markov chain. 
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1 QQ 


Table 4-5. All Pilots. Tests for 


First Test 

Hq: Independent Trials 

H]_: First Order Markov 

2 

-2 £n X x .95 Conclusion 

Scenario 2 

(33 States) 1778 1099 Reject 

Scenario 3 

(30 States) 1590 909 Reject 

Scenario 4 

(34 States) 1840 597 Reject 


of Chain (39 Pilots, 39 States max) 


Second Test 

Hq: First Order Markov 

Hj : Second Order Markov 

2 

-2 Jin 1 x .95 Conclusion 
906 34200 Cannot Reject 

817 25600 Cannot Reject 


673 37474 Cannot Reject 



VZT 


Table 4-6. Successful Pilots. Tests' for Order of Chain (11 Pilots, 16 States max) 


First Test 

Hq! Independent Trials 
Hp First Order Markov 

2 

-2 Jin X X .95 Conclusion 


Second Test 

Hq: First Order markov 

Hi : Second Order Markov 

2 

-2 Hn X x .95 Conclusion 


Scenairo 2 


(16 States) 

389 

261 

Reject 

226 

3740 

Cannot Reject 

Scenario 3 

(10 States) 

182 

103 

Reject 

60 

877 

Cannot Reject 

Scenario 4 
(15 States) 

296 

229 

Reject 

121 

3067 

Cannot Reject 



Table 4-7. Non-Private Pilots. Tests for Order of Chain (22 Pilots, 23 States max) 


First Test Second Test 


Hq: Independent Trials 

H 0 i 

First Order Markov 

H^: First 

Order Markov 

Hi: 

Second Order Markov 

-2 &n X 

2 

X .os Conclusion 

-2 

2 

Jin X x Conclusion 


Scenario 2 


(22 States) 

777 

491 

Reject 

476 

9932 

Cannot Reject 

Scenario 3 

(16 States) 

456 

261 

Reject 

233 

3511 

Cannot Reject 

Scenario 4 
(21 States) 

844 

447 

Reject 

374 

8614 

Cannot Reject 
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Table 4-8. Private Pilots. Tests for 
First Test 

Hq: Independent Trials 

Hi : First Order Markov 

2 

-2 in 1 X .95 Conclusion 

Scenario 2 

(22 States) 717 491 Reject 

Scenario 3 

(16 States) 505 261 Reject 


Order of Chain (16 Pilots, 23 States max) 

Second Test 

Hq: First Order Markov 

Hj : Second Order markov 

2 

-2 ftn 1 X .95 Conclusion 
249 9932 Cannot Reject 

340 3511 Cannot Reject 


Scenario 4 
(22 States) 


660 491 Reject 


253 9932 Cannot Reject 



Table 4-9 


High Knowledge Pilots. Tests for Order of Chain (10 Pilots, 23 States max) 


First Test Second Test 


Hq: Independent 

Trials 

Hq: 

First Order 

Hi : First Order 

Markov 

Hi: 

Second Order 

X 2 .95 

Conclusion 

-2 

2 

X X .QS Conclusion 


Scenario 2 


(19 States) 

377 

367 

Reject 

140 

6339 

Cannot Reject 

Scenario 3 
(14 States) 

272 

200 

Reject 

118 

2480 

Cannot Reject 

Scenario 4 
(20 States) 

421 

406 

Reject 

110 

7418 

Cannot Reject 



Table 4-10. Summary of Search Patterns. 


39 States Max 16 States Max 23 States Max 23 States Max 23 States Max 

All Pilots (39) Successful Pilots (11) Non-Pvt (22) Pvt (16) High Knowl (11) 


Scenario 






2 (Vac Pump) 

1st Ord 

1 st Ord 

1 st Ord 

1 st Ord 

1 st Ord 


(33 States) 

(16 States) 

(22 States) 

(22 States) 

(19 States) 

3 (Mag Fail) 

1st 0rd 

1 st Ord 

1st Ord 

1 st Ord 

1st Ord 


(30 States) 

(10 States) 

(16 States) 

(16 States) 

(14 States) 

4 (Static 

1 st Ord 

1 st Ord 

1 st Ord 

1 st Ord 

1 st Ord 

Sys) 

(34 States) 

(15 States) 

(21 States) 

(22 States) 

(20 States) 


















Such a conclusion is counter-intuitive and is not supported by an 
analysis of PIP charts for individual pilots.. The .PIP charts indicate 
strong evidence of sequential information seeking with very few returns 
to previously tested states. Yet the high order Markov chain model does 
not appear to be appropriate for the data at hand. 

In retrospect one might postulate several reasons for the failure of 
the Markov model to adequately describe the observed information seeking 
behavior of pilots. 

1) Observed trajectories across many subjects cannot 
be combined. In spite of attempts to define 
homogeneous groups, one is left with the conclusion 
that individual differences are too great to treat 
observations from n subjects as though they represented 
n realizations of the same Markov chain. 

2) Sample sizes are too small. The statistical tests 
used are based on limiting distributions. The 
sample sizes ranging from 10 to 39 in the PLATO 
data are simply too small to provide an adequate 
test of the theory. 

3) The unfamiliar task of collecting individual pieces 
of information one at a time is inhibiting to the 
memory processes of expert pilots who normally 
"chunk" familiar patterns of stimuli from an 
aircraft instrument panel. Larken et al (1980) 
observed that experts are reduced to the level of 
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novices in chess playing when pieces are arranged 
in random order on a chess board as opposed to 
being in an arrangement from a game. A similar 
problem may exist in the form in which pilots 
retain knowledge in long term memory. The awkward 
isolated search may in fact destroy what otherwise 
might be a highly structured n-state process. 

Table 4-11 shows one example of a first order Markov process. The 
frequencies (both absolute and relative) shown are for the number of 
observations of inquiry i followed by inquiry j tabulated for the 22 
non-private pilot subjects who took scenario three. The high frequency 
cells are caused by combining several unique inquiries into a single 
state. For example, state 20, magneto, includes mag left, mag right, mag 
both and mag off inquiries. Hence there is a high probability that 
•successive inquiries will return .to that state. 

The next chapter returns to analysis of individual search patterns 
as opposed to the aggregated descriptions of this chapter in hopes of 
better characterizing how "good" pilots search for information. 
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13-TACH 



14-MANIFOLD 

PRESSURE 


15-MIXTURE 


17 -PROPELLER 


18 -OIL TEMP 


19-OIL PRESS 


20-MAGNETO 


21-ICE 


22-ATC 


23-OTHER 


Table 4-11. Absolute and Relative Frequencies of Single Stage Transitions for Non-Private Pilots 
in Scenario Three. (Relative frequencies representing transition probabilities are 
shown in parentheses.) 


MB 




























^ TO 
FRCM 

15 

17 

18 

1 

19 

20 

21 

22 

23 


3-SUCTION 










3-OAT 

■ 




BSBM 





4-AIRSPEED 




mss 






10-PITOT HEAT 










11-YOKE 










12. ALT. STATIC 

- 




mu 


‘ 

• 1 


13-TACH 





HB39I 



WGK 


14-MANIFOLD 

PRESSURE 


1 

(0.091) 


i 

(0.091) 

i 

(0.091) 



m 


15-MIXTURE 

17 

(0.548) 

2 

(0.064) 

1 

(0.032) 


3 

(0.097) 



5 

(0.161) 


17-PROPELLER 

2 

(0.125) 

7 

(0.4375) 


1 

(0.0625) 


i 

(0.0625) 

1 

(0.0625) 

1 

(0.0625) 


18-OIL TEMP 

1 

(0.111) 




1 

(0.111) 

1 

(0.111) 


4 

(0.444) 


19-OIL PRESS* 

1 

(0.091) 


2 

(0.182) 


i 

(0.091) 



6 

(0.545) 


20-MAGNETO 

» 




57 

(0.890) 

1 

(0.016) 


4 

(0.062) 


21-ICE 



1 

(0.05) 


2 

(0.10) 

12 

(0.60) 


2 

(0.10) 


22-ATC 








1 

(0.5) 


23-OTHER 

7 

(0.091) 

4 

(0.052) 

6 

(0.078) 

4 

(0.052) 

5 

(0.065) 

2 

(0.026) 

1 

(0.013)' ' 

42 

(0.545) 



Table 4-11. (continued) . Absolute and Relative Frequencies of Single Stage Transitions for Non-Private 
Pilots in Scenario Three. (Relative frequencies representing transition probabilities are 
shown in parentheses.) 
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V. Evaluation of Subject Information Seeking Strategies 


By a Panel of Expert Pilots 


Recognized expert pilots were utilized in an attempt to distinguish 
patterns within past PLATO and current protocol subject data. It was hoped 
that such an analysis would further add to the descriptive model of pilot CIFE 
diagnosis and validate earlier scores on diagnostic performance. 

A. Purpose 

Six well-known aviation experts ranked two groups of ten pilots each 
on the basis of the pilots' information requests. A separate scenario 
was used for each pilot group. The purpose was to: 

a) validate a previously developed grading system that 
measured individual pilot performance on a scenario; 

b) ascertain the basis for the expert rankings; and 

c) determine if PIP templates of expert performance could be 
inferred which might lead to an optimum information seeking 
strategy. 

B. Expert Selection 

Experts chosen for the study included past and current chief flight 
instructors and department of aviation chairmen, all .from The Ohio State 
University, as well as an FAA GADO inspector. Table 5-1 shows the total 
flight hours and pilot certificates of each expert. All have their CFII 
certificate and except for one, all have an ATP certificate. 
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G. Scenario /Subject Selection. 

Subject data from two separate scenarios were chosen for 
presentation to the experts . Scenarios were selected which 
could provide wide ranges of performance in existing subject data- The 
vacuum pump failure scenario was selected because of its familarity to 
the researchers through protocol and other analyses. The static port 
blockage scenario was chosen because, like the vacuum pump scenario, it 
also provided an instrument conflict as part of the problem. 

Individual subject data were chosen for each scenario to represent a 
full range of pilot performance. For the static port scenario, at least 
one past PLATO subject was chosen to fit into each possible combination 
of correct /incorrect diagnosis and low/medium/high number of total flying 
hours. The vacuum pump scenario subjects were selected from the protocol 
analysis experiment, which at that time had 12 valid subjects. Two were 
eliminated to allow for an even mixture of correct /incorrect diagnosis 
and total number of observations. 



Table 5-1. .Expert Characteristics 

Expert 

Total Flying Hours 

Certificates 

A 

8,000 

ATP, CFII 

B 

2,500 

ATP, CFII 

C 

8,700 

ATP, CFII, A & P 

D 

14,090 

ATP, CFII 

E 

20,000 

Comm., CFII, A & P 

F 

8,000 

ATP, CFII 
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D. Procedure 

Because the PIPs preorganized each subject's information requests 
into tracks , their presentation to the experts would have likely biased 
the rankings. To eliminate this, each PIP was converted to a list of 
numbered questions showing information requests as they were typically 
asked . 

Experts were presented with a tape recorder and blank tape along 
with a packet (see Appendix F) containing: 

a) instructions which had been presented to the subjects; 

b) the vacuum pump and static port scenario descriptions; 

c) one sheet of subject information requests for each subject 
in each of the scenarios (20 sheets total with the order in 
each group of ten being determined by a table of random 
numbers ) ; 

d) the preferred diagnosis of each scenario; 

e) a complete list of responses to pilot information requests 
for each scenario; 

f) a blank ranking sheet for each scenario numbered from 
one to ten; and 

g) instructions for the expert. 

Experts were instructed to rank the pilots within each scenario from 
best (1) to worst (10). "Best” referred to a pilot whose information 
seeking behavior most clearly resembled that of an expert. In the 
process of ranking, the experts were told to describe their thoughts out 
loud, 'making certain they described their reasoning for rating one pilot 
better than another. This entire procedure was completed at the expert’s 
convenience without an experimenter present (but with the tape recorder 
running ) . 


IAS 
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Results 


Data collected from the experts consisted of : 

a) a rank sheet for each scenario; 

b) criteria used by experts in deciding on the rankings (as 
derived from the tapes); 

c) the criteria used by a particular expert. 

Tables 5-2 and 5-3 shows the subject rankings made by each expert 
for each of the scenarios. Subject 4 of the vacuum pump scenario was 
eliminated from analysis "since the experts were uncertain whether he had 
made a single lucky guess. 

For each of the two scenarios , a Spearman Rank Correlation was run 
between the ranks of each expert and each expert against the rank of 
scores assigned the subjects by the experimenters following the PLATO or 
protocol runs. 

"Within expert" agreement was very high for the static port 
scenario. Five of the 6 experts had ranks correlated with each other at 
a significance level of .05 or less. Four of the six experts were 
significantly correlated on the vacuum pump scenario. 

In terms of ranking agreement with the experimenter scores, all were 
correlated with the experimenter on the static port scenario and five out 
of six were correlated with the experimenter scores on the vacuum pump 
scenario . 

F. Criterion Selection 

Table 5-4 shows the derived criteria, which ones were utilized by a 
particular expert, and the proportion of experts making use of each 
criterion. 
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Table 5-2 . Vacuum Pump Scenario 
Subject Number Rankings 


Expert 

Rank A 

B 

C _ 

D 

E 

F 

1 

1 

13 

13 

1 

13 

1 

2 

6 

1 

6 

13 

1 

7 

3 

13 

6 

1 

7 

6 

9 

4 

3 

9 

12 

6 

10 

12' 

5 

12 

12 

7 

11 

9 

6 

6 

7 

7 

9 

12 

3 

13 

7 

10 

3 

11 

9 

12 

11 

8 

9 

10 

10 

3 

7 

10 

9 

11 

11 

3 

10 

11 

3 


Table 5- 

-3 . Static 

Port Scenario 


Subject 

Number Rankings 


Expert 






Rank 

A 

B 

G 

D 

E 

F 

1 

51 

43 

43 

51 

51 

46 

2 

43 

50 

50 

46 

50 

43 

3 

50 

51 

51 

50 

43 

50 

4 

44 

44 

46 

43 

59 

44 

5 

46 

46 

59 

54 

52 

51 

6 

59 

57 

44 

45 

44 

59 

7 

45 

45 

45 

44 

46 

57 " 

8 

54 

54 

54 

59 

54 

45 

9 

57 

59 

57 

57 

45 

54 

10 

52 

52 

52 

52 

57 

52 


NOTE: The numbers in Tables 2 & 3 are subject identification numbers 

corresponding to the files generated during protocol and PLATO 
testing respectively. 



From the recordings, a criterion was counted each time an expert either 

a) used a particular factor to differentiate subjects or groups 
of subjects from one another; or 

b) explicitly stated the criterion he was or would be using to 
make the ranking, decisions. 

Even defining the criteria selection carefully as was done, the 
frequencies remained limited on two counts. First, experts varied 
considerably in levels of analysis they applied to this study. Some experts 
would consistently give one or more reasons for each rank assigned while 
others would not give any. It was a problem of getting the expert to both 
compare in detail the subjects 1 responses and to verbalize what they 
discovered. Second, when the criterion was not explicitly stated by the 
expert, it often became necessary for the researcher to be subjective and 
infer whether or not it was a criterion. For example, one expert while 
ranking subject 46 in the static port scenario supported his rank by stating 
"he asked about ice, outside air temperature, pitot heat and opening the 
alternate static source. He suspected propeller ice when he asked about 
vibration and unusual noise." The problem here is determining whether these 
are criteria or merely summarizations to aid the expert in his recall and 
ranking of a particular subject. For the above reasons, the most accurate 
representative measure of criterion usage is the proportion of experts who 
made use of a specified criterion at least once. Thus, Table 5-4 lists the 
proportion of experts using each criterion rather than the frequency of each 
one. 
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Table 5-4. Expert Criteria Mentioned in Both Scenarios 


CRITERIA: 


EXPERT 



PROPORTION OF EXPERTS 

(+) = positive factor 







REQUESTING THE 

(-) = negative factor 

A 

B 

c 

D 

E 

F 

PARTICULAR CRITERION 

Suction gauge inquiry (+) 

X 

X 

X 

X 

X 

X 

1.00 

Alternate static port open 

X 

X 

X 

X 

X 

X 

1.00 

inquiry (+) 








Many relevant inquiries (+) 

X 

X 

X 

X 

X 

X 

1.00 

Many irrelevant inquiries (-) 

X 


X 

X 

X 

X • 

.83 

Logical sequence of inquiries (+) 

X 


X 

X 

X 


.67 

Exhibit lack of knowledge about 

X 

X 

X 

X 



.67 

aircraft systems (vacuum 
and static) (-) 








Cross-checked instruments (+) 

X 



X 

X 

X 

.67 

Random sequence of inquiries (-) 

X 


X 


X 

X 

.67 

Inquire about unknown primary 


X 



X 

X 

.5 

flight instruments (+) 








Made too few inquiries (-) 

X 


X 

X 



.5 

Did not inquire about 



X 


X 


.33 

suction gauge (-) 








Total number of inquiries 








Many (+) 


X 


X 



.33 

Many (-) 

X 

X 





.33 

Few (+) 



X 

X 



.33 

Engine power inquiry (+) 





X 

X 

.33 

Structural ice inquiry not 

X 


X 




.33 • 

made (-) 








Inquired about unknown primary 


X 





.17 

flight instruments (-) 








Too much control testing (-) 


X 





.17 

Too many separate paths taken (-) 


X 





.17 

Did not cross-check instruments (-) 





X 


.17 

Repeated inquiry (-) 



X 




.17 

Structural ice inquiry made (+) 





X 


.17 
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As critera were selected it became useful to distinguish between positive 
and negative criteria. A criterion was considered positive if the expert 
spoke of it as a factor beneficial to the subject's rating. A negative 
criterion was detrimental; that is, the subject could have performed better if 
this factor had not appeared. 

G. Analysis 

From analysis of the tapes and the criteria in Table 5-4, several 
conclusions can be drawn: 

a) Experts usually used the presence of keyword 

requests to make the initial broad cuts (i.e. top 5 and bottom 
5) in performance. The keyword was the suction gage in the 
vacuum pump scenario and the alternate static source opening in 
the blocked static port scenario. The keyword did not 
necessarily lead to a high rank. Some of the experts were more 
concerned about how the subjects arrived at the keyword (i.e. 
their information seeking strategey). 

b) Relevancy of a request seemed to weigh heavily in the experts’ 
minds yet there was a great variation among experts in what 
constitutes relevancy. 

c) Experts often categorized as "poor" those pilots who made 
random illogical requests. 
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d~) An- expert often used' a set of personal criteria which 
few others even considered. 

e) All experts appeared to be concerned with efficiency (making 
as few requests as possible and making them relevant). 

f) There appeared to be some disagreement as to whether many and 
few inquiries are positive or negative features of the search. 

Although the experts did not totally agree on a best and worst pilot for 
each scenario, there was considerable agreement if the top two and bottom two 
are considered alone. Tables 5-5 and 5-6 represent the pilot information 
requests of the best and worst subjects in the vacuum pump scenario while 
Tables 5-7 and 5-8 do the same for the static port scenario . 

H. Expert Comments 

Experts often provided useful commentaries after ranking the subjects. 

By far, the most frequent concern was the poor performance of the IER rated 
pilots. One expert stated, "I must confess at how surprised I am at lack of 
systematic search strategy or information seeking behavior exhibited by these 
pilots.... It would appear we ought to do a better job in training pilots in 
both aircraft systems and in troubleshooting when presented with information 
that is inconsistent." 
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Table 5-5. Information Requests of Two Pilots Ranked Best 
By the Experts on the Vacuum Pump Scenario 

Subject #1 

Vacuum Pump Scenario 

1. What's the manifold pressure reading? 

2 « Is there any ice on the wings? 

3. Is there any ice on the windscreen? 

4. What's the outside air temperature? 

5. Can you notice any precipitation? 

6. What's the suction gauge reading? 

7. " What's the magnetic compass reading? 

8. What's the directional gyro reading? 

Subject #13 
Vacuum Pump Scenario 

1. What are the original readings, again? 

2. What's the vertical speed indicator showing? 

3. What's the tachometer reading? 

4. What's the manifold pressure reading? 

5 . What does the artificial horizon show? 

6. What's the' suction gauge reading? 
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Table 5-6. Information Requests of Two Pilots Ranked Worst 
By the Experts on the Vacuum Pump Scenario 

Subject #10 
Vacuum Pump Scenario 

1. What happens when I open the alternate static source? 

2. What were the original indications for the airspeed indicator and 

altimeter? 

3. What's the outside air temperature? 

4. What's the tachometer reading? 

5. Is there any ice on the wings? 

Subject #11 
Vacuum Pump Scenario 

1. What's the manifold pressure reading? 

2. What's the tachometer reading? 

3. What's the oil pressure reading? 

4. What's the oil temperature reading? 

5. What's the current weather report? 

6. What is our origin and destination? 

7 . What is our terrain clearance? 

8. How far are we from any mountain ranges? 

9. What's the outside air temperature? 

10. What's the surface temperature? 

11. What's the vertical speed indicator showing? 

12. Were any thunderstorms forecast? 

13. What type of clouds are in the vicinity? 

14. What do the gear indicator lights show? 

15. What are the flaps set at? 
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Table 5-7. Information Requests of Two Pilots Ranked Best 
By the Experts on the Static Port Scenario. 

Subject #51 
Static Port Scenario 

1. What’s the forecast freezing level? 

2. Are there any PIREPS, SIGMETS, or AIRMETS in my area? 

3. Is there any ice on the wings? 

4. What's the outside air temperature gauge show? 

5. What happens when the alternate static source is opened? 

6. What happens when the pitot heat is turned on? 

7. What's the reading on the suction gauge? 

8. Is there any ice on the cowling or wings? 

9. What happens when the alternate static source is opened? 

Subject #43 
Static Port Scenario 

1. What's the reading on the vertical speed indicator? 

2. What's the reading on the altimeter? 

3. What's the outside air temperature gauge show? 

4. Is the pitot heat on? 

5. What happens when the pitot heat is turned on? 

6. What is the artificial horizon showing? 

7. What's the tachometer reading? 

8. What's the manifold pressure reading? 

9. What happens when the alternate static source is opened? 
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Table 5-8. Information Requests of Two Pilots Ranked Worst 
By the Experts on the Static Port Scenario 

Subject #52 

Static -Port Scenario 

1. Is there any ice on the windscreen or wing? 

2. Are there any PIREPS, AIRMETS, or SIGMETS in my area? 

3. What's the forecast freezing level? 

4. At what altitude are the cloud tops? 

Subject #57 
Static Port Scenario 

1. Have any circuit breakers popped? 

2. What's the suction gauge reading? 

3. What's the airspeed reading? 

4. What does the artificial horizon show? 

5. What's the altimeter reading? 

6 . What * s the turn and bank indicator show? 

7. What's the directional gyro reading?' 

8. What's the vertical speed indicator show? 

9. What's the manifold pressure reading? 

10. What's the tachometer reading? 

11. What's the exhaust gas temperature reading? 

12. What tank is the fuel selector set at? 

13. What's the oil pressure reading? 

14. What happens when the magneto switch is changed to left, right, both, and 
off? 

15. What happens when the pitot heat is turned on? 

16. What happens when the prop RPM is advanced? 

17. What happens when the prop RPM is decreased? 

18. What's the mixture set at? 

19. Is there any ice on the cowling, windscreen, wing, flap, aileron, or 
stabilizer? 

20. What is the door condition? 

21. What is the panel temperature? 

22. What is the cargo condition? 

23. What is the cabin temperature? 

24. Is there any smoke in the cabin? 

25. What's the housekeeping condition of the cabin? 

26. Are there any noticeable fluid leaks in the cabin? 

27. Is there any unusual noise or vibration in the 'cabin? 

28. What's the vertical speed indicator show? 
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Another expert commented, "This indicates that pilots, in general, have 
an almost unlimited faith in instruments when they’re flying." Commenting on 
the vacuum pump scenario in particular, this expert described the development 
of this classic situation as follows': "Even an expert pilot flying a 

single-engine airplane, primarily, with instruments vacuum powered, 

1) Is probably not aware of the failure until the instruments have 
begun to lead him astray; and 

2) When this situation does occur, almost in a classic sense the 
graveyard spiral develops and by the time a pilot recognizes 
what the problem is, the attempted recovery frequently results in 
either an unscheduled impact with the terrain or an inflight 
airframe failure." 

As a solution to this problem he suggests, "We might need to look into the 
possibility of requiring a large red vacuum system failure light or some sort 
of an annunciator that would provide the pilot, who apparently doesn't 
suspect these sorts of things, with an audible or strong visual alerting 
system." 
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I. 


Conclusions 


From the use of an expert panel we were able to conclude: 

1) There was a strong consistency among experts in their ranking of 
pilot diagnostic performance. This was especially apparent at 
the high and low ends of pilot performance. 

2 The experimenters previously developed grading scheme 

produced a ranking of subjects which correlated strongly with the 
expert rankings. 

3) Without exception, experts were concerned with the 

efficiency of subject requests (i.e. requests should be relevant and 
few in number); 

4) Ranking criteria of experts were variable except at the extremes of 
subject performance. 

5) Experts were in general appalled that IFR rated pilots could do so 
poorly in diagnostic performance. 

6) The expert ranking criteria confirmed earlier researcher conclusions 
about optimal or idealized information seeking strategies; in 
particular, the need to: 

a) confirm symptomatic problems, 

b) establish the orientation of the aircraft (i.e. was the 
aircraft in a descent?), 

c) cross check to verify instrument accuracy, 

d) use minor control inputs to check the aircraft orientation. 


157 



VI. Summary and Conclusions 


The tasks set forth in the proposal for continued work "Research on 
Computer Aided Testing of Pilot Response to Critical In-Flight Events" (Grant 
NAG 2-112) have been accomplished. This project concludes a series of 
contracts and grants (NAS 2—10047, NAG 2-75, NAG 2-112 from NASA Ames) 
directed toward a fuller understanding of how pilots collect information and 
make decisions in the face of critical in-flight events. 

The effort reported here focused on developing models of pilot decision 
making. 1) New data were obtained from some twenty instrument rated pilots by 
a lengthy protocol analysis. These data led to a frame system representation 
of how pilots organize a fault diagnosis task. 2) Previously collected 
computer data were used to describe collective behavior in terms of script 
norms, distance between inquiry measures and Markov processes. 3) Recognized 
expert pilots were used to rank both new and old subject performances and to 
suggest a rationale for good information seeking strategies. 

By-products of this research effort over the duration of the three grants 
include 3 M.S. theses, 1 Ph.D. dissertation, 4 conference presentations and 3 
journal articles. Additional publications are anticipated. Appendix G lists 
these research outputs found in the open literature. 

A. Frame System Representation 

A frame system has been developed which can account for the. performances 
of twenty pilots in diagnosing a vacuum system failure in a light aircraft. 
Eighteen frames, all having a common structure are necessary to explain the 
data. These frames represent prototypical states of nature. Each frame. 
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with the exception of the memory error frame, has an associated set of 
triggers,, a label .and two slots. 

These frames are organized into a set of hierarchies with seven 
high-level frames. All other frames are linked to these high-level frames as 
slot-fillers in "Causes" slots. High level frames are activated as a pilot 
listens to the scenario. Activation typically uses only part of the 
information available in the scenario. The information selected for attention 
appears to be caused by 1) occurrence of unexpected events, 2) priming, or 3) 
frame-directed attention. 

Pilots who identified the problem as an artificial horizon malfunction, 
but did not reach the root cause of vacuum pump failure, failed to check for 
possible causes at the appropriate time. Pilots who never reached a 
conclusion failed to ask about expected instrument readings. Those 
experiencing difficulty in solving the problem often suffered from memory 
distortions and activation of default values. 

By comparing strategies of successful and unsuccessful pilots 
recommendations for a high-level control structure to address the vacuum pump 
failure problem can be made. The proposed control structure consists of an 
ordered list of objectives along with a set of plans designed to attain these 
objectives. The plans act upon the contents of three knowledge-bases and a 
working memory. The knowledge bases include 1) the eighteen frames identified 
among the test subjects, 2) a set of objectives and plans for attaining them 
and 3) .the pilot’s aviation knowledge and episodic memory of aviation 
experiences. The working memory stores the pilot's episodic memories of the 
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scenario, the sequence of activated frames, the objectives achieved, the 
information requested and a record of the currently activated frame- Such a 
normative model could be realized as an expert system. 

Pilots who fail to diagnose the problem appear to have the necessary 
frames at their disposal, but their high-level control processes deviate from 
the suggested normative model. Their processes failed to recognize the 
objectives of: 

1. Avoiding memory distortions 

2. Detecting instrument malfunctions 

3. Avoiding incorrect activation of a frame due to inattention 
or use of insufficient information 

4. Avoiding false inferences due to activation of incorrect 
default values 

5 . Avoiding cognitive narrowing . 

B . Grouped Data 

Three different approaches were investigated in the search for 
descriptions of how pilots as a group search for information in critical 
in-flight events. These included script norms, distance measures and Markov 
chains . 

Script norms were developed by tabulating distributions for the number of 
items of information requested by varying number of subjects. The general 
shape of the distributions was the same for all five scenarios tested, with 
all but one showing a sharp difference at the 20 percent point. A group’s 
script norm was arbitrarily defined as those items requested by more than 
40 percent of the subjects. 
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From analyzing these norms it is apparent that group scripts are highly 
scenario dependent with traditional instrument clusters, showing up as high 
frequency items (e.g. oil temperature, cylinder head temperature and oil 
pressure were all sought by nearly every subject in scenarios ■with engine 
health overtones). It is also apparent that pilots as a group agree on the 
importance of a relatively small number of information items in problem 
diagnosis. In the four principal test scenarios the number of items requested 
by over 40 percent of the subjects ranged from 7 to 13. These are only small 
percentages of the 110 items of information available to CAT subjects. 

Clusters of information defined by short distances (i.e. number of 

inquiries separating the acquisition of two pieces of information) and high 

(§) 

frequencies of occurrences were identified for past PLATO subjects in both 
the vacuum pump and magneto malfunction scenarios. The purpose of this study 
was to sharpen the definition of "tracks" used in earlier flow-graph charts 
(PIPS) of pilot information seeking strategies. 

By examining the resulting tables it is possible to identify popular 
hypotheses about the system failure and the information considered important 
to test those hypotheses. The clusters for the more difficult problem (vacuum 
pump failure) often contain logically disjoint information reflecting a mix of 
hypotheses as subjects jump around in search of supportive information. The 
clusters in the magneto drive gear failure scenario may be more indicative of 
the true closeness in memory structure among items. 
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Markov chain, representations of several groups of pilots were attempted 
with limited success. Although groups were selected according to what 
appeared to be criteria related to homogeniety among members , the anticipated 
multi-stage dependence among inquiries could not be substantiated by the 
data. For the groups of pilots and scenarios tested, no stronger dependence 
than that of a first order Markov chain could be statistically identified. 

The failure of the Markov model to adequately describe observed 
information seeking behavior is believed to be related to three possible 
causes. 1) Individual differences among subjects are too great to permit 
grouping observations as though they represented multiple realizations of the 
same Markov chain. 2) The unfamiliar task of collecting information one piece 
at a time is inhibiting to the memory processes of expert pilots who normally 
"chunk" familiar patterns of stimuli from an aircraft instrument panel. 

3) The sample sizes are inadequate to test the model. 

C . Panel of Experts 

Six well-known aviation experts were used to rank two groups of ten 
pilots each on the basis of the pilots’ information requests. A separate 
scenario was used for each pilot group. The groups were selected to include a 
mix of pilot experience levels and diagnostic performance. In addition to 
ranking pilot performance each expert was asked to verbalize the reasons for 
placing each pilot in his relative group position. 

The expert rankings were highly correlated with each other as well as 
correlated with earlier experimenter derived performance scores. Without 
exception, the experts were concerned with efficiency of information requests 
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(i.e. requests should be relevant and few in number.) The derived ranking 
criteria used by this group of experts confirmed earlier researcher 
conclusions about idealized information seeking strategy; in particular the 
need to 

1) confirm symptomatic problems 

2) establish the orientation of the aircraft 

3) cross check to verify instrument accuracy 

4) use minor control inputs to check aircraft orientation. 

D. Epilogue 

The study of pilot response to critical in-flight events has been a 
rewarding experience for these researchers. We feel that we have developed a 
better understanding of the way pilots do and should organize their 
information seeking tasks during problem diagnosis. Our major contributions 
have been to document a variety of descriptive models, develop- several useful 
research tools and accumulate a large data base of pilot performance data for 
future study. 

Comments by subjects and our panel of experts plus experimenter 
observation lead to the uneasy observation that IFR rated pilots generally do 
rather poorly in diagnostic performance. The subject pilots were the first to 
admit this. Many expressed the feeling that, although humbled by the 
experience, participating in any of our experiments should improve their 
future diagnostic skills if only by reminding them of the li m its of their 
knowledge. None thought the exercises, beginning with paper and pencil tests. 
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running through GAT simulations, to computer aided tests and finally protocol 
testing, were unrealistic. All seemed to agree that such tests were valuable 
learning experiences. 

The nature of subject comments and their demonstrated performance point 
to what may well be the most useful future course for this research. The 
research tools which have been developed and tested could easily be adapted to 
the pilot training environment. Future support will be sought to accomplish 
that goal. 

It is also apparent that many pilots have weaknesses in their diagnostic 
skills which could be allievated by on-board computer systems . The 
development of expert systems to be used in a real-time setting could greatly 
improve fault diagnosis performance by pilots’. Future research directed 
toward such an electronic co-pilot is in order. Information concerning the 
memory structure of pilots modeled in this research should provide a natural 
lead-in for that research. Support will also be sought for that effort. 
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Appendix A 


Illustrations of PLATO® Displays 

The attached exhibit depicts a small sample of the material 
presented to the subject pilot by the PLATO® terminal. The 
displays selected for presentation here represent different 
facets of the program, e.g. sample biographical questions, 
sample knowledge test questions or a representative diagnostic 
scenario. The illustrations represent a sample of those 
presented to the subject in his response to: 

1) fifteen biographical questions 

2) twenty knowledge questions 

3) six scenarios 
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ORIGINAL PAGE !§ 
OF POOR QUALITY 
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IMTRQDUCT I DM 


Thank you -for being a subject in this 
supported research project. 


NASA 


The scenarios vcu are about tc'sss were 
developed net as a device to test your flying 
expertise, but rather as a mechanism Tor us 
to better understand how pjLl.o_t= .nsi-cht react 
t'b certain situations. 


ror menv 
t i u.sii *<= 


o-f the flying situations 
in real life), there are 
What we want to -find out 


p P S S 0 P; 1 5 O 

no obvious 
i s how you 


approach problem s'olvmq. These are not games 
in the sense that vcu compete with the 
computer or anvone else. We hops vou will 
•Sind the scenarios to be realistic situations 
a pilot must occasionally confront. For the 
most part, the scenarios are not dynamic 
35, the instrument panel doee not reflect 

rjycr- -r 1 In 5f*ft3Ct "tHS t&Pnjin&I SC t 

net 1 1 [:= b. flight si mui atcr but rathar as a 

dSVl CE to n{C0cari t i P; -f P ;T10 1 1 *3 P; SC* W9 Can 

or ct -p q q i X ot d i seines bid a.o d dsc i s i c*p« 
i i p; g dshs */ i op „ 


ORIGINAL PAGE m 
OF POOR QUALITY 
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ORIGINAL PAGE IS 
OF POOR QOALITt 


B I OGRftPH I CRL DRTR 


Touch the screen anywhere to begin the biographical survey. 





ORIGINAL PAGE If 
OF POOR QUALITY 


Quest ion No. 1 

Enter Pilot Certificate by Touch Panel 


a) 

. Student P i 1 ot 


b) 

Private Pilot 


c) 

Cornrnerc i a 1 Pi 

lot 

cD 

Rir Transport 

Pilot 


When you have made your- final SELECTION: 

ENTER 
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ORIGINAL PAGE If 
OF POOR QUALITY 


Quest ion No. 2 

Enter Airman Ratings Held by Touch Panel 


a) 

Repa i rman 

8) 

ASEL 

W 

i 

A i r f name Meehan i c 

kT 

AMEL 

3 

Powerp 1 ant Meehan i c 

i) 

Rotary Wing 

dj 

F 1 i ght Eng i neer 

j) 

Inspect i on Author- 
ization 

None of the above 

3 

Instrument Rat i ng 

k) 

f) 

Cert i f i ed F 1 i ght 
Instructor 


FINAL f i na 1 i zes 
above entr 1 es . 


FINAL 


ERASE removes 
last entry. 


ERASE ■ 
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ORIGINAL PAGE if? 
OF POOR QUAUTY 


KNOWLEDC 


2E SURVEY 


Touch the screen anywhere to begin the knowledge survey. 
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ORIGINAL PAGE 11 
OF POOR QUALITY 


Quest i on No . 1 

What is the standard adiabatic lapse rate? 

2 °F per 1BBB feet. 

b) 2.5°F per 1 2S&S2 feet. - 

c) 3*F per 1 ZZZ feet.- 

d) 3.5°F per 1S28S feet. 

e) 4 °F per l£r#0 feet. 



When you have made your final SELECTION: 

ENTER 
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Quest ion No. 2 


ORIGINAL PAGE -S 
0* POOR QUALITY 


Do the indications of a normally operating 
alternator system change during the course of a 
flight? (Assume charge-discharge ammeter) 


a) 


Yes: 


Ammeter shows more charge when electrical 
equipment turned on. 



Yes: 


Ammeter shows less charge when electrical 
equ i pment turned on . 


c) After engine start, the ammeter shows a higher 
than normal rate of charge and gradually' 
dec 1 i nes t o norma 1 rat e , 


d) No, does not change. 


When you have made your final SELECTION: 


ENTER 
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ORIGINAL PAGE W 
OF POOR QUALITY 


1 'diagnostic scenario TEST “1 


Inst ru ct i o ns : 

You. have a maximum of' 4 minutes for each 
d 1 agnost i c scenar i o . Si nee t base ar e 
potential emergency situations, please 
answer the question as soon as you feel 
that you have a solution. 

Please press CONTINUE when you are ready 
to start the test. 


CONTINUE 
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ORIGINAL PAGE ® 
OF POOR QUALITY 


Breaker 

/MAG^N 

i 

com 

COM2 

Trans 

Panel 

\COMP J . 


1 

pond 





Swch Heat 


if 

Yoke 



Gear Flaps Throt Prop 

RPM 


Mi* 


Alt 

Static 
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ORIGINAL PAGE E9 
OF POOR QUALITY 


Breaker 

/mag 'N 

com 1 


Trans 

Parel 

\C0 MP/- 

| 

COM2 | 

■ pond 






Left 


Right 

Fuel 


Fuel 

Qty 


Qty 




Mstr Pitot 
Swch Heat 




Gear Flaps Throt Prop Mix 


RPM 
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ORIGINAL PAGE f§ 

OP POOR QUALITY 


•Information for External Conditions 



Cow 1 i ng 
Condition 



U1 i ndscreen 
Cond i t i on 



Wing 

Cond i t i on 



Flap 

Cond i t i on 



Ri leron 
Cond i t i on 



Stabi 1 izer 
Cond i t i on 
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ORIGJMAL PAGE © 
OF POOR QUALITY 


Information for Inside Cabin Conditions 


Cargo 
Cord i t i on 



Door 

Condi t ion 



ff ane 1 T ernp 
Cond i t i’on 


Cab in T ernp 
Cond i t i on 



Housekeep i ng 
Cond it i on 


Smoke 



Fluid Leaks 



Noise & 

V i brat i ori 


Fluid Leaks: 

oil droplets on floor 


Time: 3: l# Scenario: 2\ 



Instr 
Pane 1 
RTC 
Info 
Ext 
Info 
Scen- 
ario 
GIVE 
RNSR 


L83 
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ORIGINAL PAGE IS| 
OF POOR QUALITY 


Instr 
Pane 1 


E>rfc 


Seen-* 


ini 

GIVE * 

Info 


ario 



ANSR 



atc 

Info 





m m Info 


You have chosen GIVE ANSWER. 

If you are ready to give your 
diagnosis of the scenario, 
please touch the GIVE ANSR 
button; else touch an alternate 
button to continue the test. 


Time: 1:57 Scenario: 81 
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ORIGINAL PAGE m 
OF POOR QUALITY 


1 

ail eron 

26 

elevator 

2 

alternator 

27 

eng i ne 

3 

alt i meter 

28 . 

exhaust 

4 

baffle 

29 

fai lure 

5 

battery 

3 2 

f i Iter 

6 

belt 

31 

fire 

? 

b looked 

32 

flap 

8 

bottom 

33 

flow 

9 

broken 

34 

fouled 

12 

burst 

35 

frozen 

11 

cable- 

36 

fuel 

12 

cap 

37 

gasket 

13 

carburetor 

38 

gauge 

14 

C/S fuse 

39 

gear 

15 

cock 

4 2 

govenor 

16 

complete 

41 

gyro 

1? 

condenser 

42 

heat 

18 

contro 1 

43 

hot 

19 

cold 

44 

hydrau lie 

22 

cow 1 i ng 

45 

ice 

21 

cranksha ft 

46 . 

ignition 

22 

cyl inder 

47 

i nduced 

23 

door 

48 

i nduct i on 

24 

drive 

49 

i nstrument 

25 

electrical 

52 

j ets 


51 

1 and i ng 

76 

rudder 

52 

latch 

77 

screen 

53 

1 eak i ng 

78 

screw 

54 

left 

79 

seal 

55 

1 ine 

82 

seizure 

56 

loose 

81 

smoke 

5-7 

loss 

32 

start er 

58 

lost 

33 

star vat io 

59 

magneto 

34 

static 

62 

mixture 

35 

stuck 

61 

motor 

36 

suet i on 

62 

oil 

87 

switch 

63 

part i a 1 

33 

tank 

64 

pedal 

89 

temp. 

65 

piston 

92 

throttle 

66 

pitot 

91 

tip 

67 

plugs 

92 

top 

68 

points 

93 

vacuum 

69 

popped 

94 

valve 

72 

port 

95 

vapor 

71 

pressure 

96 

v i brat i on 

72 

prop 



73 

pump 



74 

right 



75 

ring 




oil 

1 eak i ng 


STORE 

ANSWER 


gauge 


ERASE 

WORD 


line 


1 

2 

3 


n t 

P 

4 ' 

5 

6 

2 


7 

8 

9 


/ 
£ 
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ORIGINAL PAGE IS 
OF POOR QUALITY 

Quest i on No . 2 

How long do you think the airplane would fly? 


a) 

0 - 

5 minutes 

b) 

5 - 

30 minutes 

c) 

as 

long as fuel 


permits 


When you have made your final SELECTION: 

ENTER 
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ORIGINAL PAGE KT- 
OF POOR QUALITY 


Quest i on No . 3 


How critical do you judge this problem to be? 

. Cl is the least critical and 7 is the most critical) 


a) 1 


c) 3 

d) 4 

e) 5 

f) 6 

g) ? 


b) 


When you have made your final SELECTION: 

| ENTER 
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ORIGINAL PAGg M 
OF POOR QUALITY 


Quest i on No , 4 


How confident are you about your diagnosis? 

(1 is the least confident and 10 is the most confident) 


a) 1 

f) 

6 

(SI 

S) 

g) 

7 

c) 3 

h) 

8 

d) 4 

i) 

9 

e) 5 

j) 

’ 10 


When you have made your final SELECTION; 


ENTER 
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Appendix B 


Description of Diagnostic Scenarios 
and Their Answers for Scenarios 1, 2, 3, 4, and 6* 


^Scenario 7/5 is found in Figure 7 
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C-'b 


PRECEDING PAGE BLANK NOT FILMED' 




You. are making a day trip from 
Albany, NY to Bur 1 i ngton., VT . You 
fly out o f Albany at 9 : .S' ©Urn , cleared 
Victor- 9 i, Burlington. You climb to a 
cruising altitude o'f 7 0'fiTflf ft . After 
20 rn i nut es o f rout i ne If 1C f 1 y i ng 
you not i ce the sme 11 of eng i ne oil. 

How would you diagnose the problem? 


ORIGINAL PA@E Ilf 
OF POOR QUALITY . 
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Our diagnosis of the problem mas the following : 

R small crack developed iri the oil line 
feeding the oil pressure gauge. This 
crack reduced the oil pressure reading 
drastically, but did not seriously 
affect the actual i'ubr'i cat i on of the 
engine. R small pool of oil began to 
form on the floor of the cabin, pilot's 
side. Rssurn i ng that the cracked line 
would not deteriorate quickly into a 
complete break, you were in no immediate 
danger of engine seizure. 


ORIGINAL PAGE W 
OF POOR QUALITY 


CONTINUE 


191 



ORIGINAL PAGE IS 
OF POOR QUALITY 



You a re making a day trip from . 
Augusta, ME to Lebanon , NH. You fly 
out of Augusta at 9 : .Qf.ST am, cl eared 
Victor 3 9 to Meets intersect ion, 

Victor 49 6 to Lebanon . You clirnb to 
a cruising altitude of 6 MB' ft. After 
IS minutes of routine IMG flying m 
i nstrument cond r t i ons , your i nst rument s 
indicate an increase in a . ir speed and 
stead i 1 y decrees i ng a 1 1 i tude wh i 1 e 
rna i nt a i n i ng 1 eve 1 flight at 1 1 i t ude . 

How would you diagnose the problem? 
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Our d i agnos i s o f t he pr ob 1 ern ma s t he f o i I ow i ng 

Your vacuum pump failed as indicated by 
the low reading of the_ suction gauge. 

The vacuum pump drives the attitude arid 
d i rect i ona 1 gyros . Fts the art i f i o i a 1 
bor.i con. .Lost its -drive i*t -started' to sag 
to the right and you compensated by 
turn i ng 1 e f t , 1 eve 1 i ng the art i f i c i a 1 
horizon and putting the plane in a slow, 
descending left bank. The airspeed in- 
crease was due to the slight nose-down 
attitude. 


ORIGINAL PAQE ® 

OF POOR QUALITY 


iONTINUE 
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ORIGINAL. PAGE ig 
Of POOR QUALITY 

You are making a day trip from 
Keene , MH to Montps 1 i er_, VT . You 
fly out of Keene at 18 '. 3 % am, cleared 
Vi ctor- 151 to Mont pel ier . You c 1 imfo 
to a cruising altitude of 5888 ft. 

After 20 minutes of routine cruise m 
IMC your engine suddenly starts 
running extremely rough, shaking the 
whole plane and losing about 28 % of 
its cruise power. 

How would you diagnose the problem? 
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Our d i agnos i -5 of the problem mas the following : 

Your engine suffered -a broken drive 
gear in the right magneto. The 
r esu 1 1 ant unt i med i gn it i on con if 1 i ct ed 
with t he rerna i n i ng good i gn i t i on and 
caused, the extreme-l-y rough engine' and 
back f i r i ng . Sw i tch i ng from ' both ' to 
the left magneto would have resulted in 
a smooth running engine with slightly 
less power than normal cruise. 


ORIGINAL PAGE IS 
OF POOR QUALITY 


CONTINUE 


195 





ORIGINAL' PACC M 
OF POOR QUALITY 

You are making a day trip from San- 
fordjME to Messena, NY. . You fly ou.t 
o f San f ord at S : 3 0am , c 1 eared V i ct or - 
49 6 to Lebanon , V ictor- i 41 to Messena 
You climb to a cruise altitude of 6000, 

Rfter 20 min IMC flying, Boston 
Center instructs you to climb and main- 
tain i 0 , 000 ft . You acknowledge and begin 
your climb between layers, Rfter 2 min 
o f c 1 i mb , y ou notice your i nd i cat ed air- 
speed drcpp i Tig off st ead i 1 y f rom 1 00k t s , 
maintaining constant pitch attitude. 

How would vou. diaenose the oroblem? 

t * 
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Our diagnosis of the problem mas the following 


As you climbed through 6 5.0'# ft, the 
static port froze over as the outside 
air temperature dropped below 3 2 * F . 

This caused the airspeed indicator to 
decrease as altitude increased' and the 
VS I and altimeter to read lorn. 

Several corrective actions were 
possible; return to your previous alti- 
tude of 60.0’,0'ft; open the alternate 
static source; break the VS I glass. 
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:onti 


NUE 
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Seen an o 


You are makina a dav trip from 
Mcntpei ier. VT t a E'anacr , HE with 
two pasHsngsrs on board- You fly out of 
Montpelier at 1:00pm, cleared radar 
vectors to Wvlie intersection, direct 
Auousta. Victor 3 to E'anqor. You 
climb to a crux sing altitude of 9000ft, 
After 30 minutes of routine flvinc in 
instrument conditions wi tn light to 
moderate turbulence, one of your 
passengers resorts smelling a faint 
burning odor. You are unable to detect 
the odor because you have a head cold. 

What is the first thing vou would do? 
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T l me : 


Scenario: 06 


Instr 
Panel 
Xnt 
Inf o 
E>: t 

Jn-f Q 

ATC 
Inf c 
GIVE 
AWSR 
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T I N U E 


Our diagnosis of the problem was the -following 

Rear seat carpeting was smoldering. 

The rear seat passsnosr lit a 
cigarette shortly aftsr takeoff. Wh 
he disposed of it in the ashfcrav, i 
not completely extinguished. The 
cigarette fell down from the ashtray 
and was beginning. to char uphoTstery 
material. The fire was easily 
extinguished, once recognized and posed 
no i /rimed i ate danger to the flight. 
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Appendix C 


A Sample of Sub; 


iect Data 


PRECEDING PA'GE BLANK NOD FILMED 



TEST vers i on 


ORIGINAL PAGE fi 
OP POOR QUALITY 



Thi-s program ' reports the data col lected 
by the CRITICAL IN-FLIGHT EVENT program. 

Each display is a record of responses 
given by a student for each phase of the 
CRITICAL IN-FLIGHT EVENT program. 

F unct i on keys prov i ded ; 

CONTINUE will advance to the next display 
REVIEW will return to the previous display 
MENU will access the main menu display 
RESTART will start the program again 


Please enter the student that you wish to view. 
CFor example: student*J41) 

student B7 5 


Press NEX I when entered. 
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B i ograph i ca 1 Survey 



NRf'tb. : student.07 5 

DRTE: £6/09/82 
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DATA DISPLAY 


NAME: student S' 7 5 

**ci fe2 

Knowledge. Survey 

DATE: #6/09/32 


QUESTION 

(*= i rccmecfc] 

ANSR 

'AREA 


QUESTION 

[x= i rvsorreet] 

ANSR 

AREA 

* 1* 


3 


y. 

1 1 

b 

1 

2 


2 



12 

b 

3 

« 3 


2 


X 

13 

a 

1 

4 


2 


X 

14 

; 

3 

X 5 


1 


X 

15 

a 

2 

x 6 

c 

2 


y. 

16 

b 

3 

x 7 

a 

1 


X 

17 

b 

1 

S 

a 

3 


X 

13 

o 

3 

9 

b 

2 


x 

19 

d 

1 

10 

3. 

2 


X 

20 

d 

1 


SCORE = 30% 


AREA iOTAL total 

MISSED IN AREA 

1) Engine and fuel systems 

7 

7 

2) Electrical systems and 
cockp i t i nst rurnent at i on 

3 

7 

3) Weather and IFR operations 

4 

6 


CONTINUE 


■815181— 


REVIEW 


ORIGINAL PAGE 11 
OF POOR QUALITY 


204 



















TIME ATIME 

Ls.ec) Chgc) 


DISPLAY 



Ext 

info 

Ext 

info 

Ext 

info 

Ext 

info 

Ext 

info- 

Ext 

i n fo 

Int 

info 


PTC info 
PTC info 


PTC info 
PTC info 


PTC info 
PTC info 


PTC info 


ITEM 


a.i leron 


flap 
cow 1 i ng 


w i ndscreen 
wins’ 


CURRENT 

VALUE 




alt static closed 
itot heat 


pitot heat on 
- - i tot heat o f f 


f rees i ng 1 eve 1 
cloud tops 


cei ling 
visibi 1 it 


PIREPS 

SIGNETS 


PIRMETS 




CONTINUE 


REVIEW 
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DATA DISPLAY 


NAME : 

student.075 

**ci fe3 

Diagnostic Scenario #02 

DATE: 

06/09/82 

** scenes 2 





1) LEXICON RESPONSE: static - 

ice 


ORIGINAL PAGE W 
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2) FLYING TINE LEFT: as long as fuel permits 

3) HON CRITICAL PROS (1-7) : 5 

4) HOW CONFIDENT OF OWN DI AG Cl -10): 5 

5) FLYING TINE LEFT (with our diag) : 5-30 minutes 

6) HOW CRITICAL PROD (with our diag) : 5 


CONTINUE 


MENU 


REVIEW 
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DATA DISPLAY 


NAME: 

students 7 5 

**ci fe4 

Destination Diversion Test 

DATE: 

#6/09/82 


Qft/G/Afjqt. 

° F POOR 


PAGi* |§j 

Quality 


Based on the information you. have received 
so far, would you normal ly attempt this flight? 


YES 


CONTINUE 


■SlatglM 


REVIEW 
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DRTR display 

#*ci fe4 


NRI1E : student 0 7 5 

Destination Diversion Test DRTE: 0 6 / 0 9/ 8 2 


ATIME 

£s f^ hIRPORT 





INFO QUERIED 









approach aids ( 4) 


RTC serv i ces C 5 ) 


approach aids (43 
RT C serv i ces ( 5 ) 


cei l ing (2) 
visibility (3) 


IJ2F atime’ 

I is ^ RIRPGRT T>fr . r . 

l INFO QUERIED 




CONTINUE 



REVIEW 
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DATA DISPLAY 


NAME: 

student# 7 5 

*#ci fe4 

Dest i nat i cn D i vers i on T est 

DATE: 

#6/09/32 


student 07 5 " has chosen airport" *1 
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CONTINUE 

■ 

MENU 


REVIEW 


209 













DATA DISPLAY 


NAME : 

student#/ 5 

ci fe 

Previous CIFE Question 

DATE: 

#6/#9/82 

quest i on 





Have you ever had a CIFE in any of the areas? 


Electrical 

Ice 
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CONTINUE 


MENU 


REVIEW 
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Appendix D 


Combined Destination Diversion and Diagnostic Scenario 
' A Sample of Computer Displays 


211 





ORIGINAL PAGE If 
. OF POOR QUALITY 


TEST version 



We are now go i ng to present to you some 
Cr itical In-FIi ght Event s requ i r i ng your 
diagnosis of the problem. 


Assume that you are flying a fuel- 
injected Cherokee Arrow (Ni 23B) with the 
f o 1 1 ow i ng per f ormance spec i f i cat i ons : 

Cruise Speed = 135 KTAS (6 5% pwr. /» 7 S'## ft.) 

Fuel Flow (65% pwr.) = 10 GPH 

Usable Fuel Capacity = 46 gallons 

Endurance - 4.3 hours (no reserve; 

Range » 648 nautics.1 miles (no wind, no reserve) 
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original page is 

OF POOR QUALITY 


Consu 1 1 attached s i mp 1 i f i ad 1 ow a 1 1 i tu.de chart . 

You are on an IFR flight from Utah Municipal. Airport 
to Haven County Airport. You. depart on V - 1 1 JZf at 6000 ft 
in your Cherokee Arrow (N123B) which is equipped 
with a 3 -ax is autopilot. There is a NOT AM out which 
reports that Colorado VOR is out of service during 
the period you plan to navigate. Navigate using Qhigh 
and Calif orn i a VCRs . You have been enroute 60 minutes 
from Utah Municipal Airport. You are on the gauges 
but t he r i de i s smoot h . Weather hr ief i ng i nd i cat ed 
that winds at 6030 were expected to be light .and 
var i ab 1 e . 

You have one passenger aboard. 

Weather at : 

Haven County Airport = 20095 & 5 
Oh igh= is 03 & 3 

Wind Falls** i 000 & 3 by a C-172 
C US' rn i nut as ago) 

C leve Center calls and reports radar contact is lost. 

P 1 ease report present pos i t i on . 


Cervix. 

to* 

When ready, press the 

CONTINUE 

button to go 


to the VOR displav to 

as tab 1 is} 

*i pos it i on , 
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OC I — 111 o 
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DG 


HERDING 

SELECTED 



VOR 1 


CBS 



RLTITUDE 

SELECTED 



6888 



VCR 2 




D 

r* 

c 

A 

C 

T 

X 

V 


AUTO 



VOR 1 VOR 2 CTRL 

S 

HH 


r □ □ 


DD 

iff 



i .... "j 

. SELECT a device above 

■i*i 

3 

You may change OSS via 
the box next to the VOP 
or via the keypad to the 
left when ACTIVATED. 

| 

St 

D 

BB 

Time: | Scenario: 85 



. Manipulate the 
I i instruments as 
; necessary, and 
I press CONTINUE 
when through. 


I 
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POSITION REPORT 




Please report your posit ion 'by pressing 
the TO or FR buttons" for the VQR of 
your choice. (Choose at least two VQPs) . 

Type in your position via the- keyboard 
at the given arrow, then press the 
NEXT key to enter it. 


Cai i forria 
VOR 


40 ok 

rrtisBg 

Hill 

| 

Utah MunPir 
VOR 


■ Kill 

26 0 ok 

Oh i *h 
VOR ! 

jjj 


ijlll 



Press CONTINUE 
after you have 
made your report 
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RTC Response: 


ORIGINAL FAG^ ^ 
OF POOR QUALITY. 


Last C 1 earance ! 


N1C3B, thanks for the position report. 
Here is your new clearance: 

proceed direct California VOR direct 
Haven County Rirport at b.yujZL 


There w ill be oppos i t e tra f f i c 
at 5388 * . .maintain 6 tf£f>T „ 


Please confirm your new heading 
and altitude after your turn. 
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I ! 

1 SCENARIO CHANGE 


While practicing hand flying 
with your autopilot disengaged, 
you. notice that increased nose-up 
trim is required to maintain a 
constant indicated altitude and 
that your IAS has decreased 2.0'kts. 
from normal cruise. 

Your passenger notes this problem, 
and suggests that you turn back 
to Utah Municipal. 


Determine the nature of the problem, 
and your dest i nat i on dec i s i on . 
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Break-s?' 
Pan-5 1 
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RTC respons e : 

Unders t arid dec I ar i ng 
emergency enter heading 
or altitude chanee 
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P 1 ease enter your destination decision 
via the keyboard. 

home port ok 


Please enter the estimated time to vour destination 


a) 

3 - 30 

b) 

30 - 60 

c) 

60 - 90 

d) 

greater 


minutes 
minutes 
m i nutes 

than 90 minutes 


When you. have made ;,-'oar final SELECTION: 
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1 

ail erori 

26 

e 1 e vat or 

51 

1 and i ng 

7 6 

ring 

2 

al ternator 

2 7 

eng i ne 

52 

1 eak i ng 

77 

rudder 

3 

altimeter 

23 

exhaust 

53 

lean 

78 

screen 

4 

ba f f 1 e 

29 

fa i 1 are 

54 

left 

79 

screw 

5 

battery 

30 

f i 1 ter 

55 

1 ine 

80 

* < 
ITS 

v 

u;i 

6 

belt 

31 

fire 

56 

1 oose 

81 

seizure 

7 

b 1 ocked 

32 

f 1 ap 

57 

lost 

82 

smoke 

S 

bottom 

33 

flow 

58 

1 ow 

83 

starter 

9 

broken 

34 

fouled 

59 

magneto 

84 

starvat i on 

10 

burst 

35 

frozen 

6 0 

mixture 

85 

stat i c 

1 1 

cab i n 

36 

fuel 

61 

motor 

86 

structural 

12 

cap 

3 7 

gasket 

62 

none 

87 

stuck 

13 

carburetor 

38 

gauge 

6 3 

oil 

88 

suet i on 

14 

C/6 fuse 

39 

gear 

64 

partial 

89 

switch 

■15 

complete 

40 

governor 

65 

peda 1 

90 

tank 

16 

condenser 

41 

zyr-o 

66 

p i at on 

91 

temp . 

17 

contro 1 

42 

heat 

67 

p i tot 

92 

throttle 

18 

co-1 d 

43 

h i gh 

68 

plugs 

93 

tip 

19 

cow ling 

44 

hot 

69 

points 

94 

top 

2.0' 

crankshaft 

45 

hydrau 1 i c 

70 

popped 

95 

vacuum 

21 

cy 1 i nder 

46 

ice 

71 

power 

96 

va 1 ve 

22 

don * t know 

47 

ignition 

72 

pressure 

9 7 

vapor 

23 

door 

48 

i nduct i on 

73 

prop 

98 

vibration 

24 

dr i ve 

49 

i nstrument 

74 

purnp 

99 

wing 

25 

electrical 

5.0 

j ets 

75 

right 




don ' t know 


STORE | 
RNSldEB j 


ERASE | 
kJQRP j 
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Appendix E 


Destination Diversion Test : 

A Sample of Computer Displays 


ORIGINAL PAGE &§ 

OF POOR QUALITY 


PRECEDING PAGE BLANK NOT FILMED' 




original PAGE F0 

OF POOR QUALITY 



Exhibit .1 is a simplified weather chart of the North- 
eastern and Northcentral United States, and Southeastern 
and Southcentral Canada. The hypothetical flight we will 
consider will take place in the area surrounded by the 
dashed lines. You can see this area includes Vermont, 

New Hampshire, and parts of Maine, Massachusetts, New 
York, and Quebec. 


CONTINUE 
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Exhibi-t 1 


IFR conditions prevai 1 over most of our area of concern, 
except over northeastern New York, where conditions are 
slightly better. More detailed weather information will 
be provided when appropriate. 
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SCE NARIO 


You are at the Bangor International Airport .in 
Bangor,Naine, and desire to fly to Glens Falls, New 
York, for a 1:00 p.m. business meeting (shown in Fig. 

ID . The current time is 9:00 a.m. and you feel you can 
be ready for departure by 10:00 a.m. after you conduct 
all necessary preflight activities. ' 

The plane you will be flying today is your company-* s 
Cherokee Arrow (N8086W) . You have flown this 
particular plane several 'times before and regard it as 
a reliable airplane. A brief list of the important 
performance figures and IFR equipment on board is shown 
in Table I. 


1 CONTINUE 


REVIEW 
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ORIGINAL PAGE IS 
Op POOR QUALITY 

' TABL E I 

I mpor t a n t Sp ecs . a nd Performance Figures 

Cruise Speed = 135 KTAS (65% pwr. $ 78S3S3 ft.) 

Fuel Flow (65% pwr.) = \2 GPH 

Usable Fuel Capacity = 48 gallons 

i 

Endurance = 4.8 hours (no reserve) 

Range = 648 nautical miles (no wind, no reserve) 

I FR , .Equi pm e nt on Board 
2 NAV/COMMs 
2' VOR/ILS indicators 
1 RDF 

1 Three- 1 i ght marker beacon rece i ver 
1 Transponder (not encoding) 

1 Single axis autopilot 


CONTINUE 


REVIEW 
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ORIGINAL PA@I if 
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The ai.rcra-f-t- ’s fue-l tanks are- full, arid after 
a very thorough pre flight inspect ion, ' you conclude 
that it is operat i ona 1 1 y and legally ready for the 
f 1 ight . 

Now your attention turns to the weather and filing 
a flight plan. You cal 1 'the nearest Flight Service 
Station on the telephone and obtain the weather infor- 
mation in Table II. 


CONTINUE 

i.n , . , u ■■■«■■■■■ ■ i d 


REVIEW 
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TAB LE IX 

for Glenn Falls (New York) : The weather is currently 
"1002 feet overcast and 3 miles visibility in rain. “ 

It is forecast to stay that way until 1:00 p. m. , 
local time, when it should improve to 1500 overcast 
and 5 miles visibility. 

for Bangor (Maine) i The weather is currently " 1000 
feet overcast and 3 miles visibility in rain and fog." 
It is forecast to remain unchanged except for a chance 
of 500 feet' overcast and 1 mile visibility in rain, 
drizzle, and fog. 

for Albany (New York) : The weather is currently n 1000 
feet overcast and 4 miles visibility in light rain." 

It is forecast -to remain the same until 1:00 p.rn. , at 
which time it should improve to "1500 feet overcast 
and 4 miles. 

Winds aloft: from the southwest (200°) at 30 knots at 

all altitudes up to 9000 feet. 

Icing Level: 10,000 feet 

No PIREPs reported 
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TYPE 

2. AIRCRAFT 

3. AIRCRAFT TYPE/ 

imsm 

ID. 

SPECIAL EQUIP. 

IFR 

NSJ38 6W 

PA 23R-2J3r0/ T 

DVFR 




5. DEPARTURE 

6. DEPARTURE 

PT. 

TIME 


BGR 

Prop. 
12: 08 

Ac-fe. 


6 . ROUTE OF FLIGHT 

V3 to Augusta VOR V39 to Nests intersect ion V4S6 
to Glenn Falls 


9. DESTINATION 

GFA 

(Glenn Falls) 


12 . FUEL ON BOARD 


HOURS 

4 


MINUTES 

50 


13 . E5T. TIME ENRGUTE 


HOURS MINUTES 

2 15 


13. ALTERNATE AIRPORT £S3 14 . PILOT’S NAME 

A 1 bany 



15. NUMBER 
ABOARD 
1 


Id. COLOR OF AIRCRAFT 

Red on White 


CLOSE VFR FLIGHT WITH. 
ON ARRIVAL 


FSS 
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original pass ® 

. OF POOR QUALITY 



You were cleared to the Glen Falls airport "as filed". 
You lifted off from Bangor at 10:00 a.m., and your 
departure was routine. fit 10:14 (14 minutes after 
departure) you reached your cruising level of 3000 feet 
and were established on V3 northeast of the Augusta 
VOR. 


CONTINUE 
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original rase m 
of poor quality 



Rt 11:21 Ci hoar 21 minutes after departure) you cross 
Grump intersection. One minute later you hear a short 
static noise over your radio speakers. Rt the same 
time you notice you VOR needles and their "on-off" 
flags flicker unsteadi ly and return to normal 
indications. 


CONTINUE 


REVIEW | 
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ORIGINAL PAGE EH 
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Curious to know what caused these events, you glance 
over the instrument panel and find a "zero" reading on 
the ammeter. You actuate the landing light and notice 
no change in ammeter indications. From this 
information you conclude the al ternator has fai led . 


CONTINUE 


REVIEW 
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You follow the procedures in the manual but your 
attempts to bring the alternator back into service are 
unsuccessful . Therefore, you turn off the alternator, 
minimize the electrical load, and operate solely on 
battery power. 


continue! 
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ORIGINAL. FA@g Egf 
OF POOR QUALITY 



The battery, by itself, car) supply the required power 
to operate your radios for only a limited ti-me. The 
amount of time you have depends on the size and 
condition of the battery, and the power requirements of 
the essential electrical equipment you use. Even under 
ideal conditions battery power is not expected to last 
longer than 50 _ minutes. 


CONTINUE 


REVIEW 


239 








original pm: m 

OF POOR QUALITY 



You are at an altitude of 8000 feet, just west of Grump 
intersection. The time is now 11:23 and you have been 
air-bourne for 1 hour and 23 minutes. Winds are out of 
the southwest at 30 knots. 


CONTINUE 


REVIEW 
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The following information is available from air traffic 
control one piece at a time: 


1) 

Searing & Distance 


Approach Aids 

2) 

Ce i 1 i ng 

5) 

AT C Serv i ces 

3) • 

Visibility 

6) 

Terrain 


CONTINUE 


REVIEW 
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Time: 12:12 You have 'requested the following informal: ion: 


Bir- 

port 

Bearing; 

Distance 

Cei 1 

Visi 

Rpproach 

Bids 

BTC 

Serv i ces 

T erra i n 

1 







2 


700 

1 




3 

330 ° 60 

1000 

3 

VOR 

TUIR (PO 

HILLY 

4 







5 







6 

• 






7 ■ 


500 

2 




3 







9 

2JSFj0f° 25 

500 

1 

NDB 

FSS 

HILLY 

10 


500 

1 




n 







1.2 







13 

040 ° 70 

1000 

2 

ILS 

TWR (R) 

LEVEL 

14 







15 

- 






16 











• 




BIRFORT 


i 

2 

I 3 

n 

5 

6 

I 7 

8 

9 

10 

1 1 1 

12 

13 

14 


1 6 


SELECT an airport then touch ENTER. 

You will be able to fly to that 
airport and shoot one app?~oach only. 

PURPORT: "3" 

ENTER 
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RANKING EXERCISE INSTRUCTIONS 

You have just finished choosing an airport to divert to 
in the face of a serious problem,. Now we would like 
you to consider yourself to be in that same situation 
again. The next display will present a table of 
airports and descriptions in terms of RTC services, 
weather, the flight time from your present position to 
the airport, and the approach facilities there. 

We would like you to rank these airports from your 
"most preferable - Cl") to " 1 east pre f erab 1 e “ C"16 n ), 

given the same situation. Recall that you have, at the 
very most, 52 minutes of battery time left. 

You will use the touch screen to input your airport 
selection and assign it a rank. You will be able to 
edit your ranking at any time. When you have ranked 
all 16 airports ( M xi thru xl6") you will be asked if 
you want to submit the list or continue editing it. 


CONTINUE 
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ORIGINAL 
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Port 

A TC 

Serv i ces 

Cei 1 

Visi 

Time 
Cm in) 

Approach 

Aids 

Port 

RANK 

mm 

TWR (R) 

1 000 

3 

15 

ILS 

Xl 

\ 

x2 

TWR 

1000 

3 

15 

ILS 

x2 

2. 

x3 

TUJR CR) 

500 

1 

15 

ILS 

X3 


x4 

TUJR 

500 ■ 

1 

15 

ILS 

x4 


x5 

TUIR (R) 

1000 , 

3 

30 

ILS ' 

x5 


x6 

TWR 

1000 

3 

30 

ILS 

x6 


x7 

TUJR (R) 

500 

1- 

30 

ILS 

x7 

6 

x8 

TUJR 

500 

1 

30 

ILS 

x8 


x9 

TUJR 

1000 

3 

15 

NDB 

X9 

■ 

xi0 

TUJR 

1000 

3 

15 

NDB 

Xl0 


xll 

TWR CR) 

500 

1 

15 

NDB 

xl 1 


xl 2 

TWR 

500 

1 

15 

NDB 

xl 2 


xl 3 

TWR (R) 

1000 

3 

30 

NDB 

xl3 


xl 4 

TWR 

1000 

3 

30 

NDB 

xl4 

% 

xl 5 

TWR CR) 

500 

1 

30 

NDB 

Xl 5 


xl 6 

TWR 

500 

1 

30 

NDB 

xl 6 











If you knew airport "x2" had maintenance facilities, 
would you "pass up" airport "xl" for airport *x2“? 
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Expert Pilot Experiment 
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Pilot Response to Gritical-In-Flight Events 
Performance Ranking 

Thank you for taking the time to rank the problem diagnosis strategies of 
our test subjects. You will have the opportunity to rank order the 
performance of ten subjects on each of two different problem scenarios. 
Descriptions of the information provided to the subjects, problem scenarios 
and our analysis of the problems are given below. Your ranking of subject 
performance will be based on the items of information each subject requested 
to solve his problem. An ordered list of inquiries for each subject on each 
scenario is attached. An Appendix which lists our responses to each subject 
query is included for your reference. 

Please rank order these 10 performances from best to worst. By "best” we 
mean the pilot whose infomration seeking behavior most closely resembles that 
which an expert pilot should exhibit. Give the "best" pilot rank 1 and the 
worst rank 10. 

As you are .rank ordering these performances, please describe your 
thoughts out loud (with the tape recorder on). Be sure to describe the 
reasons for rating one pilot better than another. 

Please do not write on this booklet. Score sheets for each scenario 
which you may use as support documents to accompany your tape recording will 
be provided. 
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Subjects (IFR Pilots) have been given the following instructions: 

What we are going to do is have you describe what you think you as a 
pilot should do to determine the cause of a problem that has developed while 
flying a Cherokee Arrow, which is a 200 h.p. fuel-injected Lycoming. This 
particular plane is not supercharged and does not have an autopilot . 

Your purpose is to describe how a person should go about diagnosing the 
problem, not to describe what remedial actions he should take. Do not tell us 
how to correct the problem. Simply try to determine what is causing it. We 
do not want you to fly the plane, we simply want you to try to diagnose the 
problem. 

If you think the pilot should gather some particular information, you can 
ask for it and we will give you that information. For example, if you were to 
ask whether communications radio number one is operating okay, we might tell 
you that it is operating normally. You can ask for any information that might 
normally be available when you are up in a Cherokee Arrow. That includes such 
things as instrument readings, information that you could obtain visually 
outside, or information you could obtain over the radio. 

A few things to keep in mind: First we are going to give you a scenario 

and then we will be giving you any information you request. The conditions or 
instrument indications supplied all refer to a fixed point in time. Even if 
you ask a question 10 minutes after we have started, it still refers to the 
same point in time as when the scenario was read. The plane is not continuing 
to fly. 
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Second, you are allowed to make control movements but we will only give 
you the immediate response to that movement.- You will not be allowed to fly 
the plane. You cannot ask 

"If I make this change, what will it indicate 5 minutes from now?" 

We will simply tell you the immediate response of the instruments to that 
particular movement. 

Third, another constraint is that you cannot ask about the effect of 
combining 2 control movements. You will only be allowed to adjust one control 
at a time and see the response to it. You cannot adjust 2 controls at once. 

Finally, we do not want you to try to correct the problem. Instead, we 
want you to simply concentrate on determining what the cause of the problem 
is . 

Remember, what you want to do is tell us what a pilot should do in order 
to determine the cause of a problem that has developed while flying a 
fuel-injected Cherokee Arrow. - This particular plane is not supercharged and 
does not have an autopilot. Once we start, it is important that you not stop 
until you either diagnose the problem or decide that with the available 
information you cannot determine the cause. So there are essentially 2 
endpoints. Either you say "I believe the problem is such and such," or "given 
the available information, I don’t think it is possible to determine the 
cause." 
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Here is the scenario: 


Imagine that this pilot is making a day trip from Augusta, ME to Lebanon, 
NH. He flies out of Augusta at 9:00 a.m., cleared Victor 39 to Neets 
intersection, Victor 496 to Lebanon. He climbs to a cruising altitude of 6000 
feet. 

After 15 minutes of routine flying in instrument conditions in the 
clouds, the instruments indicate an increase in airspeed, a steadily 
decreasing altitude and zero pitch. 

So, the instruments indicate an increase in airspeed, a steadily 
decreasing altitude and zero pitch. 

How should this pilot go about identifying his problem? 

Our Diagnosis of the Problem for the First Scenario Was the Following : 

Your vacuum pump failed as indicated by the low reading~of 'the suction 
gauge. The vacuum pump drives the attitude and directional gyros. As the 
artificial horizon lost its drive it started to- sag to the right and the pilot 
compensated by turning left, leveling the artificial horizon and putting the 
plane in a descending left bank. The airspeed increase was due to the slight 
nose-down attitude. 

Subject Response to First Scenario 

The information requests made by 10 IFR pilots after hearing this vacuum 
pump scenario are enclosed. After each such request sequence, the pilot 
stated his conclusion. 


251 




Please rank order the performances for the vacuum pump scenario now. Be 
sure to comment on your reasons for rating one pilot better than another. 

Here is our second scenario : 

You are making a day trip from Sanford, HE to Messena, NY. You fly out 
of Sanford at 8:30 a.m., cleared Victor - 496 to Lebanon, Victor - 141 to 
Messena. You climb to a cruise altitude of 6000. After 20 minutes IMC 
flying, Boston Center instructs you to climb and maintain 10,000 feet. You 
acknowledge and begin your climb between layers. After 2 minutes of climb, 
you notice your indicated airspeed dropping off steadily from 100 Kts., 
maintaining constant pitch attitude. 

How would you diagnose the problem? 


Our diagnosis of the second problem was the following : 

As you climbed through 6500 ft., the static port froze over as the 
outside air temperature dropped below 32°F. This caused the airspeed 
indicator to decrease as altitude increased and the VSI and altimeter to read 
low. Several corrective actions were possible: return to your previous 

altitude of 6000 ft.; open the alternate static source; break the VSI 
glass. 

Subject response to the second scenario : 

Information requests made by 10 pilots after hearing this scenario are 
enclosed. After each such request sequence, the pilot stated his conclusion. 
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Please rank order the performances for the static port problem now. Be 
sure to comment on your reasons for rating one pilot better than another. 
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Work Sheet for Ranking Subjects 
(To accompany remarks on tape recording) 


Scenario : 


RANK SUBJECT // COMMENTS 


(Be sure to think out loud as 
you are trying to rate the 
subjects. ) 


1. ( "best" ) 

2 . 

3 . 

4 . 

5 . 

6 . 

7 . 

8 . 

9 . 

10. ("worst") 
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Appendix F-l 


Vacuum Pump Scenario 

Responses given to subjects when 
an item of information was requested 
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Vacuum Pump Scenario Responses 


ADF: 368 MHZ 

260° RB 
Test normal 

Aileron conditon: Clear of ice 

AIRMETS: None 

Airspeed: 145 Kts. and increasing, cruise is about 135 Kts. 

Alternate Static: Closed 

If opened no change in any instrument readings 

Altimeter (non-encoding): 5600 ft. and decreasing 

Ammeter: +2 amps (normal) 

Artificial horizon: Level, no movement 

Cabin temperature: 70°F 

Cargo condition: secure 

Circuit Breaker Panel: All Breakers closed 

Cloud tops: Area forecast 14,000 Ft. 

C0M1 : Normal 

COM2 : Normal 

Cowling Condition: Clean and Secure 

Cylinder Head Temperature: 375°F (Normal) 

DG: 250° and Steady 

Door Condition: Secure 

EGT: 1300°F and Steady (normal) 

Flap condition: Clear of ice 


Flap position: 

0° 



0°; 

0° 


10°; 

10° 

if reset to 

20°; 

then 20° 


30°; 

30° 
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Vacuum Pump Scenario Responses 


Freezing level: Area forecast 7,000 ft. 

Fuel Flow: 8 gal. /hour (normal) 

Fuel Quantity 

Left; nearly full 
Right; full 


Fuel Selector: Left Tank 

Change to left or right: no change in instrument readings 

Gear: gear up lights on- 

If lowered down; gear down lights, airspeed decays 
Groundspeed (ATC) : 120 Kts. 


Magnetic Compass : Rotating through 230° 


Magneto : Both 

left; 
right ; 
off; 
both; 


50 RPM drop 
75 RPM drop 
engine quits 
normal operation 


Manifold pressure: 20.5 inches (normal) 


Master switch: ON 


Mixture Control: Leaned for cruise 

if enrich; EGT drop, engine rough 

if lean; EGT rise, engine rough 

Noise and Vibration (cabin): Normal 

Oil pressure: 40 p.s.i. (Normal) 

Oil temperature: 140°F (Normal) 

Outside air temperature gauge: 37°F 

PIREPS: None 

Pitot Heat: Off 

If turned on: surge in ammeter discharge then back to normal 

no effect other instruments 
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Vacuum Pump Scenario Responses 


Prop RPM: 2300 RPM (nromal cruise setting) 

if increase; 2700 RPM, MP decreases 
if decrease: 1900 RPM, MP increases 

Season: Fall 

SIGMENTS: None 

Smoke : None 

Stabilizer condition: clear of ice 

Strobe: Off 

Suction gauge: Extremely low, near tbe peg 

Tachometer: 2300 RPM and constant (normal cruise reading, constant speed 

prop) 

Throttle: Cruise power 

If increase; increase in manifold pressure and airspeed reading, 
little change in altimeter reading 
If decrease; decrease in manifold pressure and airspeed reading, 
little change in altimeter reading. 

Note: If requested, artificial horizon remains level, turn/bank 

shows left turn ' 

Transponder: Code 4320, normal operation 

Trim: Trimmed for level cruise 

Turbulence : None 

Turn/Bank indicator: left turn 

Vertical Speed inidcator; 600 fpm down 

V0R1 : 111.4 Mhz , Flag On 

V0R2: 111.6 MHz, Flag On 

Winds aloft forecast: 3000, 6000, 9000 westerly at 10 Kts. 

Windscreen: clear of ice 

Wing: clear of ice 
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Vacuum Pump Scenario Responses 


Yoke: Neutral position 

If back pressure .applied: Airspeed indicator shows decrease, 

altimeter continues to decrease but at a reduced rate. 

If requested, artificial horizon remains level and turn/ 
bank shows left turn. 

If forward pressure applied: Airspeed indicator shows increase, 

. altimeter continues to decrease but at a faster rate. 

If requested, artifical horizon remains level and turn/ 
bank shows left turn. 

If left pressure applied: Turn to left on turn/bahk before 

turn, steeper after turn. 

If requested, artificial horizon remains level 
If right pressure applied: Turn to left on turn/bank before 

turn, turn to right after turn. 

If requested, artificial horizon remains level. 
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Appendix F-2 


Static Port Scenario 


Responses given to subjects when 
an item of information was requested. 
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Static Port Scenario Responses 

ADF : Normal 

Off; Off 

On ; normal 

Check circuit breaker; normal response 
test; normal response 

Aileron condition: normal 

Airspeed: Slowly decreasing from 100 Kts. 

Alternate Static: Closed 

open; airspeed suddenly increases 

closed; VSI reads correctly but sluggish 

Altimeter: Low, only 6300 ft. 

Ammeter: Normal 

Artificial Horizon: Normal climb 

Cabin Temperature: Normal 

Cargo condition: Secure 

Circuit breaker panel: All breakers closed 

Cloud Tops: Area forecast 14,000 ft. 

C0M1 : Normal 

off; normal response 
on; normal 

check circuit breaker; normal response 

COM2: Normal 

off; normal response 
on; normal 

check circuit breaker; normal response 
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Static Port Scenario Responses 


Cowling condition: clean and secure 

Cylinder Head Temperature: Normal 

DG: N-300° 


Door Condition: Secure 


EGT : Normal 

response 

Flap Condition: Normal 

Flaps : 0° 

0°; 

0° 

10°; 

10° 

20°; 

20° 

30°; 

30° 

Fluid Leaks: 

None 


Freezing Level: Area Forecast 7000 ft. 

Fuel Quantity: 

Left; 3/4 full 
Right; 3/4 full 

Fuel Selector: Left 

Lef t ; no change 
Right; no change 
Off; engine quits 

Gear: Up 

Up; Up 

Down; Normal response 

Ground Speed (ATC) : 100 Kts 

Housekeeping condition: No Loose items 

Magnetic compass: N-300° 

Magneto: Both 

Left; Normal RPM drop 
Right; Normal RPM drop 
Off; engine quites 

Both; Normal 

Manifold Pressure: Normal 
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Static Port Scenario Responses 


Master Switch: On 

On; on 

Off; electrical power lost 

Mixture Control: Normal 

Enrigh; Normal response 
Lean; Normal response 
Noise and Vibration: Normal 

Oil pressure: Normal 

Oil temperature: Normal 

Outside air temperature: 30°F 

PIREPS: None 

Pitot heat; no change 
on; no chnage 
off; no chnage 

> 

Prop RPM: Normal 

Increase RPM; Normal response 
Decrease RPM;- Normal response 

Signmets : None 

Smoke : None 

Stabilizer Condition: Normal 

Suction gauge : Normal 

Tachometer: Normal 

Thr o t tie : No rmal 

Increase; Normal response 
Decrease; Normal response 

Transponder: Normal 

Off; Normal 
On ; normal 

Check- circuit breaker; normal response 
Change; normal response 

Turn/Bank Indicator: Normal level 

Vertical Speed Indicator; Sluggish, low (100 ft. /min.) 
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Static Port Scenario Responses 


V0R1 : Normal 

off; off 
on; normal 

check circuit breaker; normal response 
test; normal response 

Windscreen condition: clear 

Wing condition: Light rime ice 

Yoke : Normal 

up; 
down; 
left; 
right; 


airspeed decreases faster 
airspeed stabilizes 
normal response 
normal response 
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Enclosed are the information requests made by 10 IFR pilots after hearing this 
scenario. After each such request sequence-, the pi-lots stated his 
conclusion. 

The actual problem used to generate this scenario (and the responses 
given to pilots after particular information requests) is given below. 

Our Diagnosis of the Problem was the Following : 

Your vacuum pump failed as indicated by the low reading of the suction 
gauge. The vacuum pump drives the attitude and directional gyros. As the 
artificial horizon lost its drive it stated to sag to the right and the pilot 
compensated by turning left, leveling the artificial horizon and putting the 
plane in a deseeding left bank. The airspeed increase was due to the slight 
nose-down attitude. 

Please rank order these 10 performances from best to worst. By "best" we 
mean the pilot whose information seeking behavior most closely resembles that 
which an expert pilot should exhibit. Give the "best” pilot rank 1 and the 
worst rank 10. 

As you are rank ordering these performances, please describe your 
thoughts out loud (with the tape recorder on). Be sure to describe the 
reasons for rating one pilot better than another. 
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Subject # 1 


Vacuum Pump Scenario 

1. What's the manifold presssure reading? 

2. Is there any ice on the wings? 

3. Is there any ice on the windscreen? 

4. What's the outside air temperature? 

5. Can you notice any precipitation? 

6. What’s the suction gauge reading? 

7. What's the magnetic compass reading? 

8. What's the directional gyro reading? 
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A Sample Response 


to the Static Port Scenario 
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Scenario 4 


Subject 45 

1. What happens when the pitot heat is turned on? 

2. What's the outside air temperature gauge show? 

•3. Is there any ice on the windscreen? 

4. What's the airspeed reading? 

5. What’s the altimeter reading? 

6. What's the throttle set at? 

7. What happens when I advance the throttle? 

8. What's the prop control set at? 

9. What happens when I advance the prop control? 

10. What's the altimeter reading? 

11. What’s the airspeed reading? 

12. What’s the mixture set at?- 

13. What happens when I lean the mixture? 

14. What’s the throttle set at? 

15. What happens when I advance the throttle? 

16. What’s the prop control set at? 

17 . What happens when I advance the prop control? 

18. What happens when the pitot heat is turned on? 

19- What happens if back pressure is applied to the yoke? 

20. What happens if forward pressure is applied to the yoke? 

21. What’s the throttle set at? 

22. What happens if I advance the throttle? 
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Appendix G 


List of Publications and Proceedings Resulting from This Research 


M.S. Theses : 

Flathers, George W., Jr. A Study of Decision-Making Behavior of 

Aircraft Pilots Deviating From a Planned Flight , The Ohio State 
University, MS Thesis, 1980. 

Lee, Jeffrey A. , A Decision Support System For In-Flight Emergencies: 
ACE , The Ohio State University, MS Thesis, 1982. 

Thomas , Mark E . , The Effect of Preparation on Pilot Diagnosis of 
Critical In-Flight Events, The Ohio State University, 

MS Thesis, 1984. 

Ph.D. Dissertations : 

Schofield, Jeffrey E. Aircrew Compliance with Standard Operating 
Procedures as a Component of Airline Safety , The Ohio State 
University, Ph.D. Dissertation, 1980. 


Journals : 


Gif fin, Walter C. and Thomas H. Rockwell. "Computer-Aided Testing 
of Pilot Response to Critical In-Flight Events," Accepted for 
Publication by Human Factors . 

Flathers, George W., Jr., Walter C. Gif fin and Thomas H. Rockwell. 

"A Study of Decision Making Behavior of Pilots Deviating From 
a Planned Flight , " Aviation, Space, and Environmental Medicine , 
October, 1982, V. 53, No. 10, p. 958-963. 

Schofield, Jeffrey E. and Walter C. Giffin, "An Analysis of Aircrew 
Procedural Compliance , " Aviation, Space, and Environmental 
Medicine, October, 1982, V. 53, No. 10, p. 964-696. 
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Proceedings ; 


Two papers appearing in: 

Jensen, R. S. (Editor), Proceedings of the Symposium on Aviation 

Psychology , April 21 and 22, 1981, Technical Report: APL-1-81, 

The Aviation Psychology Laboratory, The Ohio State University, 
Columbus, Ohio. 

i) Flathers , G. W. II, W. C. Giffin and T. H. Rockwell. 

"A Study of Decision-Making Behavior of Aircraft 
Pilots Deviating from a Planned Plight." 

ii) Schofield, J. E. and W. C. Giffin, "An Analysis of 
Aircrew Procedural Compliance." 

Two papers appearing in: 

Jensen, R. S. (Editor), Proceedings df the Second Symposium on 
Aviation Psychology , April 25 - 28, 1983, 'The Aviation 
Psychology Laboratory, The Ohio State University, 

Columbus, Ohio. 

i) Giffin, W. C. and T. H. Rockwell. "Computer Aided 

Testing of Pilot Response to Critical In-Flight 
Events." 

ii) Rockwell, T. H. and W. C. Giffin, "Combining Destination 
Diversion , Decisions and Critical In-Flight Event 
Diagnosis in Computer Aided Testing of Pilots." 
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